compile_commands.json – independent from cmake

For another project I need a so called “compile_commands.json”.

So, what is that? If you compile software from source, it is very likely that you have more than one source file; your build system (e.g. cmake, autotools) will take care about it and create some way to build the project files in the right order with the right libraries and include paths and so on set.

If you are using cmake, creating such a file is pretty easy:

$ build/ > cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=1 ..

More information about cmake and compile_commands: https://cmake.org/cmake/help/latest/variable/CMAKE_EXPORT_COMPILE_COMMANDS.html

But what about if you are not using cmake? Then you can use my project: https://github.com/btwotch/compile_commands

How does it work? My first try was to do a simple trick:

1. Create an executable that is called gcc, logs how it is invoked and then calls the real gcc
2. Put a link to this executable into a temporary directory
3. Prepend this temporary directory to the $PATH environment variable

I thought this is *the* solution. Unfortunately I was wrong. I tried it out with a local checkout of a random project that is using cmake to test it.

cmake configured the project Makefile in a way that the absolute path to the compiler is used. Then of course $PATH is not considered anymore and my symlink is evaded.

The solution!

How can I as a user with no root priviledges prevent a process to start the compiler directly?

First, some ideas I didn’t follow up:

• Creating an LD_PRELOAD library that is hooking into exec calls and logs the actual call. Disadvantage: it does not work on statically linked systems
• Using ptrace to intercept the actual exec kernel syscall and log that call. This also works on statically linked systems, but ptrace is slow and the implementation is not that easy.

So, what did I do instead? Containers! Or actually: Namespaces!

• Create a temporary directory for the compiler binaries
• Create a new mount namespace (more information about those: http://man7.org/linux/man-pages/man7/mount_namespaces.7.html)
• Bind mount the compiler binaries to the temporary directory; the advantage over a hard link is, that this works cross-filesystems.
• Bind mount my binary that invokes the compiler from the temporary directory and creates the log over the original path of the compiler (e.g. /usr/bin/gcc)

unshare(CLONE_NEWNS|CLONE_NEWUSER|CLONE_NEWPID);
if (fork()) {   
        int status = -1;
        wait(&status);  
        exit(status);   
}               
mount("none", "/", NULL, MS_REC|MS_PRIVATE, NULL);
mount("none", "/proc", NULL, MS_REC|MS_PRIVATE, NULL);
char mappingBuf[512];
int setgroupFd = open( "/proc/self/setgroups", O_WRONLY);
write(setgroupFd, "deny", 4);
close(setgroupFd);
int uid_mapFd = open("/proc/self/uid_map", O_WRONLY);
snprintf(mappingBuf, 512, "0 %d 1", uid); 
write(uid_mapFd, mappingBuf, strlen(mappingBuf));
close(uid_mapFd);
int gid_mapFd = open("/proc/self/gid_map", O_WRONLY);
snprintf(mappingBuf, 512, "0 %d 1", gid); 
write(uid_mapFd, mappingBuf, strlen(mappingBuf));
close(gid_mapFd);
for (const auto &compilerBin : compilerInvocations) {
        fs::path origPath = getOriginalPath(compilerBin);
        if (origPath.empty()) {
                continue;       
        }               
        bindMount(origPath, binDir.path() / compilerBin); 
}               
for (const auto &compilerBin : compilerInvocations) {
        fs::path origPath = getOriginalPath(compilerBin);
        if (origPath.empty()) {
                continue;       
        }               
        bindMount(fs::canonical(fs::path{"ec"}), origPath);
}               

“unshare” creates the new mount namespace – we also have to join a new user namespace to have root permissions in this container. To enter the new namespace we have to fork().

Then some basic filesystems are mounted (“/” and “/proc”).

In order to be allowed to map outside user and group ids into the namespace, we have to deny setting setgroups.

The next two steps are to set the user id mapping and the group id mapping by writing into /proc/self/uid_map, respectively into /proc/self/gid_map

In the following step all compiler binaries (gcc, g++, clang, clang++, c++, …) are bind mounted into a temporary directory.

Then the “ec” program is bind mounted over the original paths for compilers.

After that the process, e.g. “make” can be created.

Now all compiler invocations will be logged and in the end the log will be converted to compile_commands.json.

That’s it!

web2img

It is really easy to save a website into a png using webkit and qt.
Alternatives:

• wkhtmltopdf
• webkit2pdf

My code:
Web2IMG.hpp

#include <QtWebKit>

class Web2IMG : public QWebPage {
        Q_OBJECT

        public:
                void print(QUrl &url, QString &ilename);
        private:
                QString filename;
        private slots:
                void loaded();


};

Web2IMG.cpp

#include <QApplication>
#include <QtWebKit>
#include <QtGui>
#include <QSvgGenerator>
#include <QPrinter>
#include <QTimer>
#include <QByteArray>
#include <QNetworkRequest>

#include "Web2IMG.hpp"

void Web2IMG::loaded()
{
        QImage image(mainFrame()->contentsSize(), QImage::Format_ARGB32);
        QPainter painter(&image);

        setViewportSize(mainFrame()->contentsSize());

        mainFrame()->render(&painter);

        painter.end();

        image.save(filename);

        QCoreApplication::exit();
}

void Web2IMG::print(QUrl &url, QString &filename)
{
        QNetworkRequest req;

        this->filename = filename;
        req.setUrl(url);

        mainFrame()->load(req, QNetworkAccessManager::GetOperation);

        connect(this, SIGNAL(loadFinished(bool)), this, SLOT(loaded()));
}


int main(int argc, char **argv)
{
        if (argc != 3)
        {
                qDebug() << "usage: web2img <url> <filename>";
                return -1;
        }

        QUrl url = QUrl::fromEncoded(argv[1]);
        QString file = argv[2];
        QApplication app(argc, argv, true);

        class Web2IMG w2i;

        w2i.print(url, file);

        return app.exec();
}

web2img.pro

QT       +=  webkit svg network
SOURCES   =  Web2IMG.cpp
HEADERS   =  Web2IMG.hpp
CONFIG   +=  qt console

contains(CONFIG, static): {
  QTPLUGIN += qjpeg qgif qsvg qmng qico qtiff
  DEFINES  += STATIC_PLUGINS
}

Build:

1. qmake-qt4
2. make