My experience with Flock 2017

After attending Flock 2016, I got another chance to be part of Flock conference. This year, it took place in beautiful city Hyannis, Massachusetts, USA from 29th August to 1st September. Schedule of this 4 day conference was designed differently compared to last year. Both workshops and talks were running in parallel for the first three 3 days followed by a wrap-up session on last day.

Flock has lots of talks/workshops that are full of hot topics which are currently happening in Fedora. Being part of Fedora Alternative Architectures group, it is a good place for me to learn and understand about what is new in mainline Fedora. Also, I get to discuss with people to get an overall idea on efforts required to get those features available on different architectures like PowerPC, s390x, ARM.

This year, I attended various talks and workshops on topics related to Project Atomic, Modularity and Fedora CI. Other than that, I co-presented a workshop on Alternative Arches debugging and fixing with Dan Horak.


Day 1 of the conference started with keynote from our Fedora Project Leader, Matthew Miller . Best part of the talk was lots of graphs related to Fedora progress in various aspects. These included number of users using various latest Fedora releases, Fedora contributors activities across time, etc. He highlighted Fedora Atomic CI and Modularity as one of the two current objectives for Fedora.



Some of the interesting talks I attended were:

Factory 2.0

In this talk Mike Bonnet talked about development around several tools to make a better and stable fedora:

  • ResultsDB – gets info from autocloud, openqa and taskotron
  • WaiverDB – waiving a failed result, when we know test is failing incorrectly
  • Greenwave – correlates data from ResultsDB and WaiverDB
  • bodhi – enhancement done to incorporate Greenwave result
  • Freshmaker – triggers rebuilds of content based on events from other services in pipeline like rebuild a module when modulemd or specific file gets updated, rebuild containers when rpm gets to stable
  • Odcs – On Demand Compose Service generate repos (signed) of pre-release content for inclusion when building aggregate content(like: ostree, etc)

Multi-Arch Container Layered Image Build System

In this talk Adam Miller covered various topics like differences between base and layered container image. In Fedora, base image gets built by Fedora releng and layered image via OpenShift Build Service (OSBS). Currently, container image gets built only for x86_64 architecture. There is a new multi-arch build-system design in progress, which will have OSBS workers for multiple arches. Multi-arch images will further be available in registry which can be easily accessed by users.


New Container Technologies

Another nice talk from Dan Walsh on containers with the idea that “a container should be as generic as pdf ”. He introduced various tools being developed to achieve building generic container images. Some of the tools are:

  • skopeo  – various operations on container images and image repositories
  • Buildah – to create image without docker file. Has
  • cri- o – A daemon to ensure that kubernetes doesn’t break by providing end to end test. Later also integrated with openshift test suite
  • Kpod –  alternative of docker-cli

Automate Building Custom Atomic Host with Ansible

In this talk, Trishna Guha explained how to build your own custom OSTree and rebase an Atomic Host system with built custom OSTree. First approach involved doing all steps manually which is really tedious work. Later she demonstrated power of Ansible by adding steps involved in ansible playbook. I love ansible and its power!


Alternative Arches debugging and fixing

This session was organized by Dan Horák and me.  During this session, Dan talked about common architectures specific issues which may occur like data endianess, usage of signed/unsigned char, etc. He also talked about various resources and links available for people to help with debugging non x86_64 specific issues. One can get access to a non x86_64 machine available at Fedora wiki. Further talked about recent work started towards CI to track changes in upstream by regularly building them on required arches.


Later, I did a short walk through on an issues which I am debugging from quite sometime. This issue involves storage configuration error during anaconda install while building Atomic CloudImage locally on armhfp. It involves looking into Python code for error, followed by C Python binding code where actual problem may lie. This example was chosen to show how deep down an arch specific problem can lie and how we should step in to debug them patiently.

We further planned to work on fixing arch-specific issues which attendees might have in their packages but there were none from available attendees. Interested folks can look into slides and notes on debugging issue example and can talk to us later on IRC/ML.

Atomic Host 101

This session was taken by Dusty mabe which started with introduction on Atomic Host and followed by hands-on. Dusty did great job in distributing workshop’s prerequisites (Fedora Atomic vagrant image and local ostree repo) in advance. During the workshop, we did the following tasks which is nicely written in series of blogpost by Dusty:

I got a better understanding of Atomic Host from from this workshop. Initially faced some problems related to storage expansion of Atomic Host in vm, which got fixed with the help of Dusty.

Fedora Atomic Doc Work

This session was organized by Josh Berkus and Trishna Guha. Being interested and involved in Atomic Project, I wanted to attend this session in order to help with documentation as well. Josh and Trishna gave an overview of how these docs are organized and how to build and start adding content  here. Atomic Docs content are written in AsciiDoc and uses AsciiBinder to build and maintain documentation. Firstly, I did initial setup to build the Atomic Host doc repo locally and then picked up an issue from list which was about OSTree Compose. This first requires understanding of how OSTree compose is done, trying out local steps and then write the doc. Due to the nature of the topic, I promised Josh to work on it properly after Flock. At the end of the workshop, got a hot and spicy prize from Josh (thanks!)


Let’s create a module

This session was taken by Tomáš Tomeček on creating a module. During this session he took us through content of modulemd files. Modulemd is a yaml file which has various fields defined from the list specified here . To build your own module locally, write a modulemd spec file and then run mbs-build local. This seems very simple but it takes time to do initial setup and getting successful build done.

Wrap-up session

During last day of the conference, various attendees shared their experience on what they learnt or got done during last 3 days. Speakers talked briefly on how their session was received by attendees. This session was nice but I would prefer having a wrap-up session at the end of each day of conference rather than on last day.

What else?

Other than attending various sessions, I also took opportunity to talk with some of the people regarding problems I came across on multi-arch:

  • Talked with Adam Miller regarding status of building layered images on multi-arches. Layered images get build using OSBS which currently supports x86_64. Thanks to Adam on continuing his work to support building images for other arches as well.
  • Also, discussed with Adam Miller about possibilities on adding non x86_64 arches container images in Fedora registry. Right now, only x86_64 is possible. We had a short discussion along with Randy Barlow and it seems that it should be possible after adding a manifest file. I will be looking into video provided by Randy on scaled container registry and will work with him to make Fedora container registry mult-arch aware.
  • User Feedback on Modularity :  I also went to Modularity feedback session to see what’s going on here. I previously gave it a try when Boltron – Modular server preview release occurred. I was more interested in terms of getting it working on multi-arches. Discussed issues I came across while while trying out on ppc64le:
    • Fedora 26-modular iso boots and starts anaconda installer in text mode. Didn’t see a way to start in graphical mode.
    • Fedora 26-modular iso is not able to fetch nearest mirror during anconda installation. Even proving manual repo path, doesn’t seem to work.

Modularity team took these feedback positively and I hope that these issues will get addressed in upcoming release.

Also, I asked for possibility of getting Modular Fedora container image built for multi-arches to try out. Right now, it’s not possible because it gets built using osbs (multi-arch build support is in progress). Good thing is Tomáš Tomeček volunteered to help me with building container image locally. He will first try out on x86_64 and will share a blogpost on same.

I am looking forward to spend more time on Fedora-modular on multi-arch.

There were many more interesting sessions but I couldn’t attend due to multiple sessions in parallel. Waiting for videos from talks and workshops to be uploaded in order to watch some of them which I missed.

I fully enjoyed Flock this year – from attending sessions to meeting Fedora friends to evening Flock events to winning lot of goodies at Wackenhammer’s Clockwork Arcade and Carousel 😉 Thanks to Brian Exelbierd, rest of the organizers, volunteers and sponsors for making Flock awesome. Also, big thanks to my employer Red Hat for sponsoring my trip!

It was nice to meet you all in person, now it’s time to go back and continue working on things on fire until we meet next time 🙂


Flock 2016 & my talk on ABI checking in Fedora

Flock is the annual Fedora conference where you can find Fedora contributors as the main audience. This year the conference was held at the beautiful city Kraków, Poland from 2nd to 5th August. Being a schedule of 4 days, it was split into first 2 days of talks and later on workshops. Majority of talks were enriched with various Fedora related topics.

Main Talks

Since I have recently started to contribute to Fedora, I decided to attend only couple of talks and spent rest of time talking to more Fedora contributors. Some of the talks which I attended includes keynote by Matthew Miller on State of Fedora 2016, keynote by Radek about modern and open way of education among kids, modularity in Fedora talk by Langdon White and status of Pagure by Pierre-Yves Chibon.

28737950212_939a0dff2a_m          28558622010_c73a65833c_m

Most of talks have been recorded and  has been uploaded on YouTube. Link to recordings and slides for talk can be found at Fedora wiki.


With my interest to have patches in Kernel, I attended workshop on Building the Fedora Kernel by Laura Abbott. During this workshop I successfully built Fedora Kernel locally in fast mode using ./scripts/ x86_64 kernel_srpm. Also, learned building upstream kernel with enabling/disabling desired modules during build using command make menuconfig (Need to try out!)

I am not usually a design person but still went to try out my chances in Fedora badges workshop organized by Maria Leonova and Marie Nordin 🙂 To design badges you need to have inkscape software (dnf is our friend) installed. To use Fedora specific color, copy palettes files (*.gpl) into ~/.config/inkscape/palettes/ and get one of badges templates. Now pick up new open ticket and you are ready to go. I tried working on an opened ticket which was to create a badge for “You helped get NodeJS 4 into Fedora”. So far, it seemed easy but seriously placing NodeJs logo in the right way in Fedora template was not so easy. I tried some ways and thought it’s better I stay away from it otherwise Fedora might get some ugly badges 😉


Our (I and Dodji Seketeli) talk on Ensuring ABI stability in Fedora

Our main goal to have this talk in Flock was to make more people aware of the concept of Application Binary Interface and how one can ensure that application which they are writing/maintaining are ABI compatible across various releases. Slides and recording of our talk are also available to view.

During this talk, Dodji covered:

  • What ABI in C/C++ applications means
  • Artifacts which may lead to ABI changes like:
    • File format – right now we support only ELF file format
    • Target architecture for which binary has been compiled
    • Calling convention
    • Addition/Removal of functions and variables in source code
    • Change in type(s) of existing functions
  • Among found ABI changes in application, only some of them are incompatible changes such as: removal of existing functions or variables, incompatible layout changes in types.


Later on, I continued the talk and explained:

  • Automatic ABI checking performed in package updates shipped in Fedora koji using abicheck task integrated with taskotron.
  • Depending upon abicheck result, status message (PASSED, FAILED or NEED_INSPECTON) with log is sent to package maintainer(s) ( whoever has subscribed) for review. This is already running in production which you can view anytime.
  • Other than abicheck task run, libabigail provides tools which can be used by package maintainers (fedabipkgdiff, abipkgdiff) and developers (abidif, abipkgdiff) during development phase to avoid releasing ABI incompatible applications.
  • Verifying ABI changes log with live example taken from abicheck task run on gpgme package.
  • Various future improvements in our tooling like reducing memory consumption during ABI check run, extending ABI check run to all koji packages(right now runs only on packages mentioned in critpath), running ABI checks in rawhide on two distinct packages.


Talk was well taken by attendees and they had various curious questions (which have been recorded in our talk video).


During this conference, We managed to talk to some taskotron maintainers (Tim Flink and Kamil Páral) and got their feedback on existing status of abicheck task run. They seem pretty ok with its current status. We also discussed on following future works which needs to be done to improve ABI checking experience in Fedora:

  • Running abicheck task on all koji packages – We are already running ABI checks on important userspace packages with few exceptions due to memory constraints (kdelibs, firefox, thunderbird). Existing runs are stable now and it is time to extend to run on all packages and we all agreed in favor of. A task has already been created on phabricator which will get done soon.
  • Compare abicheck on two distinct builds in rawhide update – Currently, when abicheck run on rawhide packages, it ends up comparing between same nvr which is not very useful. This is because there is no tags like update-testing which we have in Fedora branches. In rawhide, latest build become the greatest. We thought of multiple solutions to fix it but for now we agreed to go on easily doable solution. We will compare latest build in rawhide with second immediate latest one. More details is getting tracked on phabriactor.
  • Run abicheck task only if package has Shared library – Right now abicheck task is only relevant if run on C/C++ shared libraries. Lot of packages in Fedora are non c/C++ and without shared libraries. So, it doesn’t make sense to waste resources by running on all packages. We have a task created in phabricator which will look into rpm package content first and look for shared libraries. Regex which we will use to find out shared libraries files are \.so[0-9.]*$’. If a package has at least one shared library then only perform abicheck run on it.

Overall, the conference was productive to me both technical and community wise. I met a lot of people in person to whom I have only talked over IRC channel/mailing lists. Also met some new people from the community. Thanks to all the organizers who did a great job with the organization. Everything was so great including venue, arrangements and evening events (sadly I couldn’t attend walking tour to city). I captured some pictures during my Flock trip which can be found at flickr.


Would love to visit Flock again with more contribution in Fedora 🙂