Help understanding and debugging slow boot with systemd network-online.target

Question

I've spent some time on google, but I don't want a quick fix here, I want to understand what is going on. I have a desktop which boots to Windows in 3 seconds, and booted older ubuntu's in under 10, but with Kubuntu 15.10 it takes over 40 seconds to boot, and the KDE login is also very slow.

$ systemd-analyze critical-chain 

graphical.target @9.961s
└─multi-user.target @9.961s
  └─getty.target @9.961s
    └─getty@tty1.service @9.961s
      └─rc-local.service @9.925s +3ms
        └─network-online.target @9.924s
          └─network.target @1.000s
            └─wpa_supplicant.service @1.772s +338ms
              └─basic.target @797ms
                └─sockets.target @797ms
                  └─avahi-daemon.socket @797ms
                    └─sysinit.target @795ms
                      └─networking.service @740ms +54ms
                        └─apparmor.service @416ms +323ms
                          └─local-fs.target @409ms
                            └─local-fs-pre.target @409ms
                              └─systemd-remount-fs.service @401ms +6ms
                                └─system.slice @128ms
                                  └─-.slice @128ms

So it's clear that the delay is due to waiting to get online. Many questions here:

1. Why would a desktop version of Ubuntu block my boot and log in waiting for a network connection? Can these services not start after I booted? I want to understand why anyone thinks this is reasonable at all. Even for a server, wouldn't it be better to let it boot and then start up services which require network?

2. How can I debug which services are blocking on network? I tried $ systemctl show network-online.target and it just shows me that the online target is wanted by kerneloops.service. Why would that service need internet to start? Also, this seems like something that would come standard on install. I imagine I can disable that one, but I don't want to spend the next 3 hours disabling and rebooting one by one. Is there a better way to see all of the services which are blocking on network-online.target? Can I defer these services to start post-login instead of disabling them?

3. This is the least important to me, but on a desktop with an ethernet connection to a router, why is it taking nearly 10 seconds to get online?

Answer 1

I (partially) solved this, so I wanted to post how in case anyone with similar problems finds this.

Summary: I commented out the line After=network-online.target in /lib/systemd/system/rc-local.service.d/debian.conf. Note that this may break rc.local scripts which depend on network.

Here's how I found it. First, I read some resources here.

From my post above, I knew that network-online.target was slow, and from googling I knew that it was the thing that waits to have a network connection. I ran systemctl list-dependencies network-online.target --before to determine which services were waiting until after network started, and saw that rc-local.service was there. Also, the graphical target (which I think is the thing that brings up the log in screen) depended on rc-local.service. I ran systemctl cat rc-local.service and saw that Debian had added the After dependency on online, some more info in this bug.

I think this is safe, as long as you don't have anything in rc.local that needs network, which should be the case for most normal non-server installs.

Now my system "boots" in about 3 seconds, but takes ages to log in after I type my password, so I may have just kicked the bucket down the road, but at least I made progress.