How to force openGL to draw a non-convex filled polygon - c

Question

It seems to be the standart that polygon are drawn in a convex shape. See stackoverflow.com/questions/15556929/open-gl-polygon However it is not for me if I choose it to be filled. How can I have my polygon filled while maintaining the shape I defined?
void drawFloor(){
// White
glPolygonMode(GL_FRONT_AND_BACK, GL_FILL); // GL_LINE works the way I want it too
//glDisable(GL_POLYGON_SMOOTH); // has no effect :(
glBegin(GL_POLYGON);
glColor3f(1.0, 1.0, 1.0);
glVertex3f(0, 0, 0); //0
glVertex3f(2*boxSize, 0, 0); //1
glVertex3f(2 * boxSize, -boxSize, 0); //2
glVertex3f(5 * boxSize, -boxSize, 0); //3
glVertex3f(5 * boxSize, 4 * boxSize, 0); //4
glVertex3f(6 * boxSize, 4 * boxSize, 0); //5
glVertex3f(6 * boxSize, 6 * boxSize, 0); //6
glVertex3f(0 * boxSize, 6 * boxSize, 0); //7
glVertex3f(0 * boxSize, 2 * boxSize, 0); //8
glVertex3f(1 * boxSize, 2 * boxSize, 0); //9
glVertex3f(1 * boxSize, 4 * boxSize, 0); //10
glVertex3f(2 * boxSize, 4 * boxSize, 0); //11
glVertex3f(2 * boxSize, 3 * boxSize, 0); //12
glVertex3f(3 * boxSize, 3 * boxSize, 0); //13
glVertex3f(3 * boxSize, 2 * boxSize, 0); //14
glVertex3f(2 * boxSize, 2 * boxSize, 0); //15
glVertex3f(2 * boxSize, 1 * boxSize, 0); //16
glVertex3f(0, 1 * boxSize, 0); //17
glEnd();
}
This code results in:
http://postimg.org/image/o4wt9ij33/ with GL_LINE
http://postimg.org/image/l31fltgkf/ with GL_FILL
I want only the inside of the polygon filled, keeping the shape of the version with the GL_LINE. I did not expect the output of GL_FILL to be so different.
What I want (created with MS paint):
http:// postimg.org/image/d3gbf613d/
I am using Win7+ Visula studio express2013+ Renderer Intel HD Graphics Version: 3.1.0 - Bulid 8.15.10.2509+ GLSL Version 1.40

Answer 1

Note: This answer shares content with my recent answer to a similar question here: Black out everything outside a polygon. I did not nominate the questions as duplicates because they sound different, and could potentially have different answers, even though this one happens to be mostly the same.
One approach for drawing non-convex polygons is to break them down into triangles. There are a number of algorithms that can do this, which can be found by searching for keywords like "polygon triangulation".
OpenGL has another mechanism that works great for this: stencil buffers. An explanation of this approach is in the Red Book under Drawing Filled, Concave Polygons Using the Stencil Buffer. The main idea is that one can draw a triangle fan with an arbitrary origin and your polygon vertices. The pixels that are inside the polygon will then be drawn an odd number of times, while the pixels outside the polygons are drawn an even number of times. The stencil buffer is used to track the odd/even count.
To outline the main steps:
While setting up the context and drawing surface, make sure that a configuration with a stencil buffer is requested.
During drawing, clear the stencil buffer along with the color buffer, and enable the stencil test.
glClear(GL_COLOR_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
glEnable(GL_STENCIL_TEST);
Set up state for the render pass that counts if pixels are rendered an odd/even number of times. Note that this must only write to the stencil buffer, so color writes are disabled. The key part is the GL_INVERT for the stencil op, which flips the stencil value each time a pixel is rendered, which ends up storing the odd/even count in the stencil buffer.
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glStencilFunc(GL_ALWAYS, 0, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_INVERT);
glStencilMask(1);
Render a triangle fan with an arbitrary point, e.g. (0.0, 0.0), as the first vertex, and the polygon corners as the remaining vertices. The polygon must be closed, so the first and last polygon corner must be the same. If p1, p2, ... , pN are the polygon corners, the sequence of vertices for the GL_TRIANGLE_FAN draw call is:
(0.0f, 0.0f), p1, p2, ... , pN, p1
A trivial shader can be used for this pass since the color value is not even written.
Enable color writes again, and set up the stencil test attributes to render only pixels that were rendered an odd number of times in the previous pass.
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glStencilFunc(GL_EQUAL, 1, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
Draw geometry that covers the entire area of the polygon, and possibly more. This can for example be the triangle fan from step 4, or a bounding box of the polygon. Only the part within the polygon outline will be rendered, the rest is eliminated by the stencil test.