I agree with your vision. I have tried to use those options before, but they have no affect. Maybe it was a bug and it works now?

I would say that if lxc.network.ipv4 and lxc.network.ipv4.gateway might be set for some other reason, then that will be my next step. First of all is simple addition to test gitlab and see how a commit goes.

I can change the parameters to what ever you like and perhaps only start with the two that I need. Is netmask necessary? It does seem that it would be set inside the container though.

PS. The current project that I am working on uses containers to isolate services on a network. So each container is unique. I understand the use case for dynamic provisioning, failover, or load balancing. So I will try and keep that in mind. Though, I might initially only add what is useful for me and then extend. I don't like to make huge changes all at once - otherwise I would have re-written the whole file separating everything into functions. I am not that destructive. ;)

On 06/21/2016 11:39 PM, Simon Walter wrote:

I agree with your vision. I have tried to use those options before, but they have no affect. Maybe it was a bug and it works now?

Privileged or unprivileged? Test to verify it works with your configuration. If not, figure out why before developing further because you wont be able to test.
I always test using privileged containers. That way if something doesn't work I know it's not related to using an unprivileged container.
Test with manual configurations prior to coding. Without this experience, then experience remains limited. View it as an exploration exercise.
If an edge case remains that doesn't work as advertised, then we code around it. Or better yet, see if a bug has been filed. If not, and the problem can be reproduced, then consider filing a bug report.
BTW, if you specify `lxc.network.ipv4` in a container config, and `lxc-ls -f` only shows that IP for a brief period, before it changes to something else. That's because DHCP inside the container reassigned the IP. This is resolved by changing the interface definition from `dhcp` to `manual`.
Ex. ``` auto eth0 iface eth0 inet manual ```
This is something lxc-devuan can do when assigning a static IP via --net-ipv4. And issue a message to that effect.
> I would say that if lxc.network.ipv4 and lxc.network.ipv4.gateway might be set for some other reason, then that will be my next step. First of all is simple addition to test gitlab and see how a commit goes. > > I can change the parameters to what ever you like and perhaps only start with the two that I need. Is netmask necessary? It does seem that it would be set inside the container though.
Without the netmask what's the broadcast address? What's the network address?
The netmask is *fundamental*. Without the fundamental knowledge it will be unclear how to properly configure the network options, let alone program for it.
Another way of specifying netmask is CIDR notation, i.e. x.y.z.t/24 Where /nn = number of network bits. See: https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing
Google is your friend. Find/review several LXC networking setups. Also see: man lxc.container.conf
Netmask isn't set inside the container unless someone goes in and does that. Anyone who does that is going to also configure the host IP on same interface. This leads right back to my previous comments about why that's undesirable for containers.
You won't find a separate lxc netmask option. The lxc.network.ipv4 parm uses CIDR notation to designate the netmask.

On 06/22/2016 01:07 AM, Simon Walter wrote:

PS. The current project that I am working on uses containers to isolate services on a network. So each container is unique. I understand the use case for dynamic provisioning, failover, or load balancing. So I will try and keep that in mind. Though, I might initially only add what is useful for me and then extend. I don't like to make huge changes all at once - otherwise I would have re-written the whole file separating everything into functions. I am not that destructive. ;)

I don't fault your focus on things useful to your situation. On the other hand being included as a Project Developer does imply a willingness to cooperate to achieve project goals. Otherwise why not just clone or fork and go happily down your own path?
I mainly care about logical and intuitive option/feature choices, correctness, and proper testing to ensure we don't break existing function, or add something that's easy to cause breakage by misuse.
So far testing seems lacking, as does your networking experience. Both of these can be overcome but only you yourself can make that happen. This includes researching to answer your own fundamental questions. Doing your own due diligence is *always* a minimum expectation.
Re: dynamic provisioning, failover, or load balancing: Keep in mind, many people who aren't doing those things still clone and go for their own testing. They wouldn't limit themselves either by configuring /etc/network/interfaces.
Functions are fine, but you can go overboard with them too. Overuse of functions will result in over-abstraction. Then the next person has to jump all over just to read the code.
Take my configure_locale() function as a for instance. It a nice neat small function that does only one thing. And it's intuitive what it does by its name. However it didn't absolutely need to be a separate function.
It's small enough and it's not called from anywhere else so it could have stayed as in-line code. In fact the only reason I made it a function is so I could move it around easily and get rid of those perl warnings. And I didn't even need to do that because after analyzing, I hit the right spot the first time.
So I like functions too. But please make sure there's a good reason before arbitrarily creating one.
Not making huge changes at a time is a good rule of thumb. Fortunately, and by design, this enhancement can be accomplished without re-engineering any of the existing code. Each of the five "divide and conquer" points are separate and simple to code, not one big complex glob. However there's no small way to *properly* implement this, because those five parts work together. It's either done, or it's not.
I'll keep an open mind about things though, and see what you come up with.
JFTR, I'm not totally against having the capability to configure in /etc/network/interfaces. It's just *way* more important to do it per LXC network design.
Cheers

Let me know what testing is missing. I've never done TDD with scripts, but I suppose it's a good idea. My current development platform for this template is a virtual machine running Devuan. I then create containers running Devaun. Is that any good? I suppose I should test unprivileged containers as well. I would like to test Devaun containers running in a Debian host. Debian is available as an OS option from many dedicated server shops. I imagine that I will end up using that.

What I've found after a couple hours of trying different things, and excuse me if I missed it somewhere in the docs, the order of lxc.network options is important. If you get the order wrong, nothing happens. I am not sure if this was the case in 2013, when I first used LXC. I just assumed LXC was still buggy and that wasn't really going to work anyway. So I went ahead and wrote it like I did. I will now make full use of the LXC config file and try to get things right concerning the default.conf.

Since the lxc.network options really do handle the network in the container fine , I don't think we need to set up the network in the container's interfaces file. Is there a use case you can image for that?

Simon Walter @smwltr commented

Let me know what testing is missing. I've never done TDD with scripts, but I suppose it's a good idea.
My current development platform for this template is a virtual machine running Devuan.
I then create containers running Devaun. Is that any good?

That's a perfect start. We also need to test installing Devuan from non-Devuan containers and the same approach will work fine.
So instead of creating containers running Devuan, create other containers such as Alpine, ArchLinux, Centos, Debian, Fedora, Gentoo, OpenSuse, Oracle, Ubuntu, etc..
Then we install LXC within each, turning them each into Nested LXC Hosts. Then within them `git clone` the Devuan template and use it to install Devuan Jessie/Ascii/Ceres. This will expose any weaknesses when installing from a foreign OS.
This is a lot to do for sure. So we split them between us and document the necessary actions/tweaks to turn each OS into a Nested LXC Host. That way we both will eventually end up with the capability to configure nested LXC within *any* OS.
For instance, as documented in my README.md:

LXC versions tested:

• LXC 1.1.5 on my Workstation/Laptop
  • Devuan Ceres <== 1st level
  • Devuan Jessie <== 1st level
  • Devuan Jessie (32-bit) <== 1st level
• Nested LXC 1.0.6 within Devuan Jessie container <== 1st level Jessie from above
  • Devuan Ascii <== 2nd level
  • Devuan Ceres <== 2nd level
  • Devuan Jessie <== 2nd level
• Nested LXC 2.0.1 within Devuan Ceres container <== 1st level Ceres from above
  • Devuan Ascii <== 2nd level
  • Devuan Ceres <== 2nd level
  • Devuan Jessie <== 2nd level

The foreign OS's we want to test installing Devuan from are **_new_** 1st level entries: Ex. "Nested LXC n.n.n within ForeignOS Release container"
IOW they're Nested LXC *Hosts*, and the Devuan containers created within them are the actual Nested containers.
The 2nd level entries are always Devuan.
This time we don't really care about which version of LXC is available with a given OS, that problem has been solved. We just use whatever LXC version the OS comes with.
Here's an incomplete example (showing just the packages), how I turned a Devuan container into a nested LXC host:
**LXC packages:** lxc, lxctl, redir, debootstrap, and optionally uidmap (for unprivileged, not needed yet)
**Git packages** git, git-man
**Network packages** dnsmasq, iptables, bridge-utils, ethtool
**Log packages** (syslog + lnav makes it easy to see what's going on) rsyslog, logrotate, cron, lnav
**Config info:** (TODO add config info)
The actual commands used and the net config is missing. I'll document that later.
Obviously this will take some time to pull off. However it doesn't require additional hardware. And once it's done the first time, it will be a breeze to repeat as long as we document idiosyncrasies needed for each OS.
Plus, it's a pretty cool accomplishment!
> I suppose I should test unprivileged containers as well. I would like to test Devaun containers > running in a Debian host. Debian is available as an OS option from many dedicated server shops. > I imagine that I will end up using that.
Yes. We definitely also want to test from Debian.

Let's not spend time now testing unprivileged (unless you
want to). That's an entire sub-project in and of itself. And it
may likely require template config changes. Therefore I view
this as out of scope at this time. Also it carries implications
to the Host LXC configuration which is off limits to the template.
We can always revisit the idea in the future.

> What I've found after a couple hours of trying different things, and excuse me if I missed it > somewhere in the docs, the order of lxc.network options is important. If you get the order > wrong, nothing happens. I am not sure if this was the case in 2013, when I first used LXC. I just > assumed LXC was still buggy and that wasn't really going to work anyway. So I went ahead and > wrote it like I did. I will now make full use of the LXC config file and try to get things right > concerning the default.conf.
The following option starts a new interface definition: `lxc.network.type = veth`
That's why my default.conf has this: ``` # Network configuration lxc.network.type = veth <== start of new interface lxc.network.hwaddr = 00:16:3e:6b:f7:7e lxc.network.flags = up lxc.network.link = lxcbr0 lxc.network.ipv4 = 0.0.0.0 ```
I haven't been bitten by any option order problems. I suggest retaining the order.
> Since the lxc.network options really do handle the network in the container fine , > I don't think we need to set up the network in the container's interfaces file. Is there > a use case you can image for that?
The use case I came up with is the same as how Xen used to configure eth0 as a bridge. It was hard to follow. The average user stood almost no chance of understanding how it worked.
Basic outline of what Xen did (from memory, don't hold me to this): * Down physical NIC eth0, remember IP config * Remove IP config from eth0 (leaving no level-3 config) * Rename physical NIC eth0 to peth0 (no IP config assigned, data-link only) * Create bridge named eth0 * Add peth0 as port on bridge eth0 * Assign bridge eth0 the same IP config from the original eth0 * Up peth0 and eth0 * Turn on promiscuous mode for peth0 NIC. * Add VM vif interfaces to bridge
The end result: * peth0, the physical NIC (old eth0), is the data-link * Host and VMs on eth0 bridge connect to LAN via peth0. * VMs on eth0 bridge all appear as separate machines on the LAN.
Caveat: VMs must be configured in same subnet as the Host/LAN.
In this type of setup, where you want machines visible on the LAN, then, and perhaps only then, does it make some sense to configure IP addressing in the container `/etc/network/interfaces` file. Then again, nothing stops from using the lxc config in this situation also.
So *maybe* it's a potential use case. But it's on the very edge.

I suppose I should make the command line arguments override things in /etc/ethers.used. My reasoning is that it is explicit on the command line. If someone has configured /etc/ethers.used and does not set the corresponding command line argument, that they will have the settings from /etc/ethers.used populated the container's config for that host.

What do you think? Am I missing anything?

You do understand the only setting coming from/etc/ethers.usedis thehwaddr, yes?
The README.md documents this, and how Dnsmasq uses it to assign IP, etc. How that
works is something you'll also need to come to grips with. At some point try it out.

If the design is followed as I outlined above, the CLI override happens **without** any change to existing code.
The part about the five "divide and conquer" points in my original email, included above, makes it quite clear. Specifically this part immediately following the five points:

By timing correctly and inserting lxc.network entries after the
comment "# Add net config", you won't have to worry about my earlier
code that assigns hwaddr from /etc/ethers.used. It will even retain
the comment as part of the entry value. And the hwaddr can also be
overridden by --net-hwaddr, giving all CLI options the last word.

The only thing it doesn't cover is, if someone configures /etc/ethers.used, **and** they setup Dnsmasq to assign IP from the hwaddr, **and** if 'dhcp' is still specified for eth0 in /etc/network/interfaces (because lxc-devuan didn't change to `manual` with the CLI IP override), **and** they neglect to override hwaddr from the CLI (they shouldn't have to if eth0 is configured `manual`), then and only then will dhclient request/reassign another IP. So this is unlikely to occur.
I take extra time when I write to try and be thorough and cover to a level necessary to properly communicate the complete idea. Sometimes I miss the mark, but usually not.
Please re-read the design above, and then ask for further clarification if needed.

My time is going to be limited for a while. My elderly mother, who I take care of, just had to have stents put in due to an attack yesterday. I just got the call and there was muscle damage, which makes todays prognosis considerably worse than yesterdays when it looked to be more minor. So like I said my time will be more limited. Hopefully I've already provided enough of the detail you need to move forward for now.

Hi Greg,

I hope your mother is well.

I've made quite a few changes to the template over the last month and have used it hundreds of times. Though I will not say that it's completely tested.

Since there are so many changes, and you may not like them, I don't know if I should send you a pull request or send you the template to look over before hand.

Let me know how you want me to proceed or if you are even interested in this project anymore.

Thank you,

Simon