From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from tb-mx1.topicbox.com (localhost.local [127.0.0.1]) by tb-mx1.topicbox.com (Postfix) with ESMTP id 9C2B11E4DD5E for ; Sun, 4 Aug 2024 15:31:18 -0400 (EDT) (envelope-from josh@sysmgr.org) Received: from tb-mx1.topicbox.com (localhost [127.0.0.1]) by tb-mx1.topicbox.com (Authentication Milter) with ESMTP id BE5864DA803; Sun, 4 Aug 2024 15:31:18 -0400 ARC-Seal: i=1; a=rsa-sha256; cv=none; d=topicbox.com; s=arcseal; t= 1722799878; b=t8HwGVBMx3DJUSQgb5jsmKXAxRkVQVpCipvAzkNnNJE4ENdELm 0yXszM+Nyr9uuyELmmA4902G9FbFi0eI4MRb9hYXjEuU77o1QLZj6AU99rf5duvQ L1F0eUuJQteqoOLhKcO2Bf5KP/0+HmzYv772kzSX/LthGEJAtbL/zUZfUvaIoq2+ nlw1ow2AldrnfjWENe+gey+R7OmYqYZYJM/1FtSnaw3xD+Mhf/noDp7dSL2FIQXR YLIZConFzItl8csMvZOHFyropglzVFIQSEFp7+58B3J0xXEZLuNlpjOuqxk8R0PS MaZXSLslpOjjcs8eN0GqT9XQp3PuNzgDuBrw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d= topicbox.com; h=mime-version:references:in-reply-to:from:date :message-id:subject:to:content-type; s=arcseal; t=1722799878; bh=Yq1P9Br36A53LvxjK/w5BzslSD6u0dLvwpjeRppBm80=; b=sitz0i3fFw3B rkZBLJg9PUOc/bXvhAsDbD6nUIaNzGzh4pDlZ8sr0Eq39qJ6KI+UmZdOzubEXuaw vGDk9wJRP+OsYSeoPFNyE3W+aweYSHSpr+rXXJKSMBD3/mUJ66TovRaNE205rYpL cTxvjvw4xYToSRjeZaLLO7oN/ReFGyd6mkkr2WiKcVzcw2cMwI+TtTT4DfE1YjFB Pj/hZiqfvU7OeHq+kdRXoAhoiD3MI3Ihq3taqexawThiUB+PZiz3/Z3vGA3wBqcv I+0u3szWCyYx4pp6A9ctZB6stb+3LH9hSdkG46cHN2h3vM7guFYK55v1ifKHmWNC pPsTuPE/tw== ARC-Authentication-Results: i=1; tb-mx1.topicbox.com; arc=none (no signatures found); bimi=none (No BIMI records found); dkim=pass (2048-bit rsa key sha256) header.d=sysmgr.org header.i=@sysmgr.org header.b=A5Q/+Kmq header.a=rsa-sha256 header.s=google x-bits=2048; dmarc=pass policy.published-domain-policy=quarantine policy.applied-disposition=none policy.evaluated-disposition=none (p=quarantine,d=none,d.eval=none) policy.policy-from=p header.from=sysmgr.org; iprev=pass smtp.remote-ip=209.85.167.182 (mail-oi1-f182.google.com); spf=pass smtp.mailfrom=josh@sysmgr.org smtp.helo=mail-oi1-f182.google.com; x-aligned-from=pass (Address match); x-google-dkim=pass (2048-bit rsa key) header.d=1e100.net header.i=@1e100.net header.b=g4ZIbbUd; x-me-sender=none; x-ptr=pass smtp.helo=mail-oi1-f182.google.com policy.ptr=mail-oi1-f182.google.com; x-return-mx=pass header.domain=sysmgr.org policy.is_org=yes (MX Records found: alt2.aspmx.l.google.com,alt1.aspmx.l.google.com,aspmx.l.google.com,alt3.aspmx.l.google.com,alt4.aspmx.l.google.com); x-return-mx=pass smtp.domain=sysmgr.org policy.is_org=yes (MX Records found: alt2.aspmx.l.google.com,alt1.aspmx.l.google.com,aspmx.l.google.com,alt3.aspmx.l.google.com,alt4.aspmx.l.google.com); x-tls=pass smtp.version=TLSv1.2 smtp.cipher=ECDHE-RSA-AES256-GCM-SHA384 smtp.bits=256/256; x-vs=clean score=-100 state=0 Authentication-Results: tb-mx1.topicbox.com; arc=none (no signatures found); bimi=none (No BIMI records found); dkim=pass (2048-bit rsa key sha256) header.d=sysmgr.org header.i=@sysmgr.org header.b=A5Q/+Kmq header.a=rsa-sha256 header.s=google x-bits=2048; dmarc=pass policy.published-domain-policy=quarantine policy.applied-disposition=none policy.evaluated-disposition=none (p=quarantine,d=none,d.eval=none) policy.policy-from=p header.from=sysmgr.org; iprev=pass smtp.remote-ip=209.85.167.182 (mail-oi1-f182.google.com); spf=pass smtp.mailfrom=josh@sysmgr.org smtp.helo=mail-oi1-f182.google.com; x-aligned-from=pass (Address match); x-google-dkim=pass (2048-bit rsa key) header.d=1e100.net header.i=@1e100.net header.b=g4ZIbbUd; x-me-sender=none; x-ptr=pass smtp.helo=mail-oi1-f182.google.com policy.ptr=mail-oi1-f182.google.com; x-return-mx=pass header.domain=sysmgr.org policy.is_org=yes (MX Records found: alt2.aspmx.l.google.com,alt1.aspmx.l.google.com,aspmx.l.google.com,alt3.aspmx.l.google.com,alt4.aspmx.l.google.com); x-return-mx=pass smtp.domain=sysmgr.org policy.is_org=yes (MX Records found: alt2.aspmx.l.google.com,alt1.aspmx.l.google.com,aspmx.l.google.com,alt3.aspmx.l.google.com,alt4.aspmx.l.google.com); x-tls=pass smtp.version=TLSv1.2 smtp.cipher=ECDHE-RSA-AES256-GCM-SHA384 smtp.bits=256/256; x-vs=clean score=-100 state=0 X-ME-VSCause: gggruggvucftvghtrhhoucdtuddrgeeftddrkeeggddugedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnh htshculddquddttddmnecujfgurhepgghfjgfhfffkuffvtgesthdtredttddtjeenucfh rhhomhepfdflohhshhhurgcuofdrucevlhhulhhofidfuceojhhoshhhsehshihsmhhgrh drohhrgheqnecuggftrfgrthhtvghrnhepvdehffejkeegtdejfeelveelteegveejkeeu heefjeegveekhfevffeljedtgedvnecuffhomhgrihhnpehgihhthhhusgdrtghomhdpsh ihshhmghhrrdhorhhgnecukfhppedvtdelrdekhedrudeijedrudekvdenucevlhhushht vghrufhiiigvpedtnecurfgrrhgrmhepihhnvghtpedvtdelrdekhedrudeijedrudekvd dphhgvlhhopehmrghilhdqohhiuddqfhdukedvrdhgohhoghhlvgdrtghomhdpmhgrihhl fhhrohhmpeeojhhoshhhsehshihsmhhgrhdrohhrgheqpdhnsggprhgtphhtthhopedupd hrtghpthhtohepoeguihhstghushhssehlihhsthhsrdhilhhluhhmohhsrdhorhhgqe X-ME-VSScore: -100 X-ME-VSCategory: clean Received-SPF: pass (sysmgr.org: Sender is authorized to use 'josh@sysmgr.org' in 'mfrom' identity (mechanism 'include:_spf.google.com' matched)) receiver=tb-mx1.topicbox.com; identity=mailfrom; envelope-from="josh@sysmgr.org"; helo=mail-oi1-f182.google.com; client-ip=209.85.167.182 Received: from mail-oi1-f182.google.com (mail-oi1-f182.google.com [209.85.167.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by tb-mx1.topicbox.com (Postfix) with ESMTPS for ; Sun, 4 Aug 2024 15:31:17 -0400 (EDT) (envelope-from josh@sysmgr.org) Received: by mail-oi1-f182.google.com with SMTP id 5614622812f47-3db1eb76702so6598466b6e.0 for ; Sun, 04 Aug 2024 12:31:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sysmgr.org; s=google; t=1722799876; x=1723404676; darn=lists.illumos.org; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=Yq1P9Br36A53LvxjK/w5BzslSD6u0dLvwpjeRppBm80=; b=A5Q/+Kmqfhr0CFml9tEYxaxen3ha33KvuzBEoHCnz+TCO1P+TFP6NMGuwUwvxr31Em xqJKVBygXigdrXIp/StjN/CVtqyAtKAjZTXa8S+76ZHKpqlcES8t+w6X6wfVEgKDE6qq mSTEOc50F4IW6VABkh85pfFw6qCL5yChKs/l7jAEpRJDnEE7pYh0ebwRdI+n0auBvVX6 KVwmKKLX/m4Y9vc3Fo+5Vm7fPsELHSJoWQjbTTl/QgbxAqdhXMkfeczhXktcxv5LCh0z tQb7s2vuy/491iXiZNYjOSxjRWaAnGH20Au1hHhbQRCFvsUEvSFKL7X5sOS45Tvo6M0z 1UNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722799876; x=1723404676; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Yq1P9Br36A53LvxjK/w5BzslSD6u0dLvwpjeRppBm80=; b=g4ZIbbUdhw3g3wfKiNekg/0BiX1p6FVujvWB8W+qTPdILq1FsNUvssFHiFP8PLKtlt V3l6wmI3TxSmbs5EhkyUvzO3+Q93fg99R8ck6nLtsUJEawn5GfuNNWlLxI9E0zE+iW3W VM5+gcY4mZN+QLFJh+208a6HfkQnhk3TomaAn634D2kU/XbLsUxR+PutNwLAMfK0+GDA S/nCjDdOrqFZHxBuj1oBPVHHl2p6fOG2Mq7GBKFVW41GFbumcObxI7jajMlNuYrEwNZD qKe5bzulPKMxRllUowF1XPbi/P3EaKsaWdmM1W/z7e6BS+XoC+KlcPn7ychc1/7u0NRY mV0w== X-Gm-Message-State: AOJu0YyJQw+b1V4Ij6zPzi9o3au2E2XfiroDBX+yCnEpHtC1GZSSj7I6 UNbkYNQCTc0CwiQBymU4zNPKfNg2pvnUX9LjkJeOQ4qkzVTotoGP+Wc6CnfYFjPptfJa0z34nLG jndc2Xay52+bM0OmuJp2KWjeBzpbkz+25t+8yEs9AqBh5xT3ieUY= X-Google-Smtp-Source: AGHT+IESN4cJO6ze1KFLFnOTUi9qbaIKP3kppIT+rEhetJQzPBw1X0f0gN6fl4loBJzkSUoqs2R2eZYllQoQuCnwccE= X-Received: by 2002:a05:6808:1924:b0:3d9:3f5d:7b59 with SMTP id 5614622812f47-3db55786e0fmr12835924b6e.0.1722799875915; Sun, 04 Aug 2024 12:31:15 -0700 (PDT) MIME-Version: 1.0 References: <4522967a-aac0-411d-be8a-d3358db3bb89@puptv.com> In-Reply-To: <4522967a-aac0-411d-be8a-d3358db3bb89@puptv.com> From: "Joshua M. Clulow" Date: Sun, 4 Aug 2024 12:31:03 -0700 Message-ID: Subject: Re: [discuss] A PkgSrc VM for Helios / Omicron1 To: illumos-discuss Content-Type: text/plain; charset="UTF-8" Topicbox-Policy-Reasoning: allow: sender is an admin Topicbox-Message-UUID: 21d14712-5298-11ef-9cd0-eaf3c1809a85 On Sun, 4 Aug 2024 at 08:07, d wrote: > Is it possible, and how hard would it be to make a fork of the omicron1 > branded sparse VM for Helios that supports PkgSrc? The source for the omicron1 brand is here: https://github.com/oxidecomputer/helios-omicron-brand It's not completely accurate to say that it's sparse, for what it's worth. Sparse zones currently mount some trees, like /usr, read-only from the host system. You then can't modify that tree at all; what's in the GZ version of /usr is in the zone, no more, no less. I wanted to add the ability to modify the contents of /usr, to be able to add or remove files as part of layering on additional packages -- a bit like a docker image does. We don't currently have a union file system, and in fact union file systems are complex in (correct) implementation, which is why I didn't just write one at the time. Instead, I used some important facts about the intended structure of Helios systems in the Oxide product (a ramdisk-based appliance) to make a simpler approach. At zone install time, we: - create a blank ZFS file system - unpack a baseline tar file to get a blank /etc and /var - make a recursive _copy_ of whatever happens to be in /usr and /lib in the GZ into the zone file system - unpack more layer tar files on top of the result of the previous steps, if provided Both the base ramdisk image for the host, and the baseline tar file, are assembled together from IPS packages at image build time. This means they're never out of sync with one another. Also critically, libc in the zone image is never out of date with the host kernel, even though we made a copy, because it's all ramdisk ephemera anyway; when we reboot and want to start more zones, we install them from scratch, potentially mounting in delegated datasets that have the specific persistent data we needed (like database files, etc). This presents a challenge, though, for workstations and engineering systems in the lab, where we use a classic install-to-disk UNIX system approach that allows one to add packages and mutate the system in the ways that are convenient in that setting. There, each time you "pkg update" to get new OS software, the kernel _would_ be out of sync with the libraries zone root. The baseline archive would also be out of sync, and no package update actions would have occurred. We solve the baseline issue by regenerating it in-situ using pkg(1) in a boot-time SMF service after each boot. As described in the omicron1(7) manual page, and the README, for development systems it's your responsibility to hold it correctly. This means any time you do a "pkg update" and reboot, you must also uninstall and reinstall any zones after you reboot. This will use the updated baseline, and will also re-copy all the files from /usr that must be in sync. Ultimately, you must simulate what we do on the ramdisk system for it all to hold together in the expected way. I don't have a succinct answer to the question, but there are some things that might help you answer it on your own: - the pkgsrc bootstrap is just files to lay down in the file system, so really you could probably just create a regular omicron1 brand image layer tar file that you could pass to the regular (unmodified) brand to get a zone that has pkgsrc bits inside, just as we do with Omicron components - the omicron1 brand is really, for production use, intended to be used in a very particular way, in a very particular ramdisk environment (described in the documentation and above) -- if you aren't planning on building and booting ramdisk images I'm not sure that I would use it, given the caveats for its use on install-to-disk systems - we made a conscious choice to focus on hardware-level virtualisation in the Oxide product, in no small part due to the (relative!) ease with which user workloads can be punted between machines using live migration; this is not something that's likely to be easy or even possible to achieve with zone-based workloads -- if you're planning to use our control plane, but on i86pc systems, I suspect you're going to run into more and more friction as we lean more heavily into the live migration stuff for things like seamless software update and so on. Cheers. -- Joshua M. Clulow http://blog.sysmgr.org