Check if the kernel framebuffer driver returns errors on bad ioctls

When probing for the copyarea ioctl, we want to be sure that the
kernel just does not return 0 (success) for any unsupported ioctls.
The rockchip vendor kernels have been reported to have this issue.

In the case if the support for the Raspberry Pi specific copyarea
ioctl was detected by mistake, moving windows or scrolling was
broken.
Signed-off-by: Siarhei Siamashka <siarhei.siamashka@gmail.com>

src/fb_copyarea.c

@@ -38,6 +38,11 @@
  * defined in "linux/fb.h" header
  */
 #define FBIOCOPYAREA		_IOW('z', 0x21, struct fb_copyarea)
+/*
+ * HACK: another non-standard ioctl, which is used to check whether the
+ * fbdev kernel driver actually returns errors on unsupported ioctls.
+ */
+#define FBUNSUPPORTED		_IOW('z', 0x22, struct fb_copyarea)
 
 /* Fallback to CPU when handling less than COPYAREA_BLT_SIZE_THRESHOLD pixels */
 #define COPYAREA_BLT_SIZE_THRESHOLD 90
@@ -64,6 +69,16 @@ fb_copyarea_t *fb_copyarea_init(const char *device, void *xserver_fbmem)
         return NULL;
     }
 
+    /*
+     * Check if the unsupported dummy ioctl fails. If it does not, then the
+     * kernel framebuffer driver is buggy and does not handle errors correctly.
+     */
+    if (ioctl(ctx->fd, FBUNSUPPORTED, &copyarea) == 0) {
+        close(ctx->fd);
+        free(ctx);
+        return NULL;
+    }
+
     /*
      * Check whether the FBIOCOPYAREA ioctl is supported by requesting to do
      * a copy of 1x1 rectangle in the top left corner to itself