[9fans] Upspin - a respin of 9p?

[9fans] Upspin - a respin of 9p?

[Bakul Shah]

[-- Attachment #1: Type: text/plain, Size: 2046 bytes --]

https://upspin.io/doc/overview.md

Upspin provides a global name space to name all your files. Given an Upspin name, a file can be shared securely, copied efficiently without “download” and “upload”, and accessed from anywhere that has a network connection.
....
Upspin can name information from any data service, not just traditional files.
----
Initial impression:

IMHO, its usefulness is integrating a bunch of things. A path has a global user id (user@foo.com) as a root, which is looked up in (what I would call) a root server. From it you find the directory server which stores the metadata for the remaining path. From this you find the data server where the file or data source is actually located and an ID meaningful to the server (like qid but can be a content sha1 sum). The directory server also checks if the requester is allowed access and presumably gives her a public key of the root user to be able to decrypt the data.

Clearly, if the source is not an ordinary file, there can be no sha1 sum -- presumably the directory server doesn't care.

The overview talks about the design being geared toward friends and family (ala Dropbox?) but the only thing I see that would be hard to scale is the fact a dir tree has an ACL. A dir server may also end up being a bottleneck.

User data can be protected by the owner but the dir server needs to be able to read metadata such as ACL, data location etc.

Not sure if the design allows for dynamic bind/mount. This would require a more flexible dir server structure... (I haven't read the code so this is pure speculation). But I'm wondering if something the CPU command can be implemented. May be there is a protocol to attach your own dir server.

Renames are probably not handled to avoid atomicity (just speculating). Or may depend on a dir server. 

ACLs are for dir trees. From the syntax it looks like you can add more access in a sub tree but not remove it.

I'd have preferred a capability scheme instead of ACLs -- need to think more about this.

[-- Attachment #2: Type: text/html, Size: 3097 bytes --]

Re: [9fans] Upspin - a respin of 9p?

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 545 bytes --]

I don't think it's a reworking of 9P.
It's closer to an older style of distributed file system, closer to
Amoeba's or the Cambridge Distributed System,
and using full-content storage operations on content accessed through a
separate and global name service.
9P (and relatives) allow a huge assortment of surprisingly different
service types to be represented and accessed in a uniform way,
where conventional file storage is easily the least interesting service.
(That isn't a criticism: both this and 9P-like things have their place.)

[-- Attachment #2: Type: text/html, Size: 794 bytes --]

Re: [9fans] Upspin - a respin of 9p?

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 286 bytes --]

On 23 February 2017 at 09:00, Charles Forsyth <charles.forsyth@gmail.com>
wrote:

> (That isn't a criticism: both this and 9P-like things have their place.)


And you can point the name server at other services, and store public keys
in the key server, which has possibilities.

[-- Attachment #2: Type: text/html, Size: 578 bytes --]

Re: [9fans] Upspin - a respin of 9p?

[hiro]

Wow, the post-fossil era has begun.

I'm optimistic about the aims and curious about the design.

I guess hashes would be taken before encryption and then stored with
other plaintext metadata.

Would the public be aware which files I possess if they know my mail
(by recognising the filename or the hash)?

>From what Bakul says there are some kind of servers in use (run by
third party's?). Are they trusted to encrypt things on my behalf, or
are they just receiving the already encrypted versions from me with
some amount of plaintext metadata as mentioned before?

good to see something new again.
hiro

On 2/23/17, Charles Forsyth <charles.forsyth@gmail.com> wrote:
> I don't think it's a reworking of 9P.
> It's closer to an older style of distributed file system, closer to
> Amoeba's or the Cambridge Distributed System,
> and using full-content storage operations on content accessed through a
> separate and global name service.
> 9P (and relatives) allow a huge assortment of surprisingly different
> service types to be represented and accessed in a uniform way,
> where conventional file storage is easily the least interesting service.
> (That isn't a criticism: both this and 9P-like things have their place.)
>

Re: [9fans] Upspin - a respin of 9p?

[Bakul Shah]

The overview doc says the data source can be other than a file. A camera or twitter feed for example. What seems to be missing is the namespace splicing (no bind or mount). There is also some talk of IoT sources.

Need to think through this but conceptually I don't see a problem adding this 9p feature back within the same architecture.... A thought experiment worth doing.

Even for serving named live data decoupling data source/sink from naming can have value. You should even be able to use a name to take advantage of an underlying CDN! Or a content tree ala Usenet. E.g. Not all data is directly served from your camera but from a content delivery network so scaling is not an issue.

But I need to first read and understand what all they've done!

> On Feb 23, 2017, at 1:00 AM, Charles Forsyth <charles.forsyth@gmail.com> wrote:
> 
> I don't think it's a reworking of 9P.
> It's closer to an older style of distributed file system, closer to Amoeba's or the Cambridge Distributed System,
> and using full-content storage operations on content accessed through a separate and global name service.
> 9P (and relatives) allow a huge assortment of surprisingly different service types to be represented and accessed in a uniform way,
> where conventional file storage is easily the least interesting service.
> (That isn't a criticism: both this and 9P-like things have their place.)
> 

Re: [9fans] Upspin - a respin of 9p?

[hiro]

To answer my own questions:

> I guess hashes would be taken before encryption and then stored with
> other plaintext metadata.

not really:
"The path name, revision number, encrypted content location,
signature, and wrapped keys are the primary metadata about a file
stored by the directory server."

> Would the public be aware which files I possess if they know my mail
> (by recognising the filename or the hash)?

The directory server and at least all people i share the file with
would be aware.

> From what Bakul says there are some kind of servers in use (run by
> third party's?). Are they trusted to encrypt things on my behalf

no, they only have the metadata above or encrypted data.

So, if I understand correctly if you can access the right directory
server you can find out what filenames I use and what people I share
them with.

Re: [9fans] Upspin - a respin of 9p?

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 294 bytes --]

On 23 February 2017 at 09:34, Bakul Shah <bakul@bitblocks.com> wrote:

> What seems to be missing is the namespace splicing (no bind or mount).


That's a function of the operating system (or not), even with 9P, so
nothing here rules that out.
There are some interesting possibilities.

[-- Attachment #2: Type: text/html, Size: 645 bytes --]

Re: [9fans] Upspin - a respin of 9p?

[Ryan Gonzalez]

[-- Attachment #1: Type: text/plain, Size: 2493 bytes --]

https://upspin.googlesource.com/upspin/

*looks at mascot*

Eh, Glenda's cuter. This looks like a sleep-deprived crack-addicted mobster
chick.

--
Ryan (ライアン)
Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else
http://refi64.com

On Feb 23, 2017 2:30 AM, "Bakul Shah" <bakul@bitblocks.com> wrote:

> https://upspin.io/doc/overview.md
>
> Upspin provides a global name space to name all your files. Given an
> Upspin name, a file can be shared securely, copied efficiently without
> “download” and “upload”, and accessed from anywhere that has a network
> connection.
> ....
> Upspin can name information from any data service, not just traditional
> files.
> ----
> Initial impression:
>
> IMHO, its usefulness is integrating a bunch of things. A path has a global
> user id (user@foo.com) as a root, which is looked up in (what I would
> call) a root server. From it you find the directory server which stores the
> metadata for the remaining path. From this you find the data server where
> the file or data source is actually located and an ID meaningful to the
> server (like qid but can be a content sha1 sum). The directory server also
> checks if the requester is allowed access and presumably gives her a public
> key of the root user to be able to decrypt the data.
>
> Clearly, if the source is not an ordinary file, there can be no sha1 sum
> -- presumably the directory server doesn't care.
>
> The overview talks about the design being geared toward friends and family
> (ala Dropbox?) but the only thing I see that would be hard to scale is the
> fact a dir tree has an ACL. A dir server may also end up being a bottleneck.
>
> User data can be protected by the owner but the dir server needs to be
> able to read metadata such as ACL, data location etc.
>
> Not sure if the design allows for dynamic bind/mount. This would require a
> more flexible dir server structure... (I haven't read the code so this is
> pure speculation). But I'm wondering if something the CPU command can be
> implemented. May be there is a protocol to attach your own dir server.
>
> Renames are probably not handled to avoid atomicity (just speculating). Or
> may depend on a dir server.
>
> ACLs are for dir trees. From the syntax it looks like you can add more
> access in a sub tree but not remove it.
>
> I'd have preferred a capability scheme instead of ACLs -- need to think
> more about this.
>

[-- Attachment #2: Type: text/html, Size: 3906 bytes --]

Re: [9fans] Upspin - a respin of 9p?

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 334 bytes --]

To install this distributed file system software on my Windows machine, I
need to install Go, which is too big for what remains on the built-in
drive, so I pointed it at a removable one, and the installer gives the
diagnostic "Installation directory must be on a local hard drive", which is
slightly ironic in the circumstances.

[-- Attachment #2: Type: text/html, Size: 392 bytes --]

Re: [9fans] Upspin - a respin of 9p?

[Skip Tavakkolian]

[-- Attachment #1: Type: text/plain, Size: 3509 bytes --]

To me, one of the problems Upspin tries to solve is how to establish
authorization (a.k.a ACL, entitlement, etc.) in a decentralized way, once
authentication is established.  As far as I can tell the only use of "key"
server is to verify the user's identity; but perhaps it's also used to
decrypt "directory" server content.

Regarding 9P, I've not studied Upspin source code yet, but conceptually it
would be like separating the hierarchy traversal parts of the protocol
(e.g. clone, walk, etc) from the content access part (open/read/write) and
replacing them with RPC with directory and store servers. Conventionally,
the content on store server is encrypted, so the user doesn't have to trust
the store server.

The problem I see is the user needs publicly accessible directory and store
servers if he/she wants to share content. I think having a way of providing
a namespace to a trusted server without having to figure out how to setup
secure servers is required for average users.

By contrast, 9PCloud's approach is to be the namespace multiplexer and
allow namespace sharing with authenticated users. Because all connections
are initiated from the user end, 9P exporter and importer are behind
firewalls and the content resides on the user's device.

On Thu, Feb 23, 2017 at 12:27 AM Bakul Shah <bakul@bitblocks.com> wrote:

> https://upspin.io/doc/overview.md
>
> Upspin provides a global name space to name all your files. Given an
> Upspin name, a file can be shared securely, copied efficiently without
> “download” and “upload”, and accessed from anywhere that has a network
> connection.
> ....
> Upspin can name information from any data service, not just traditional
> files.
> ----
> Initial impression:
>
> IMHO, its usefulness is integrating a bunch of things. A path has a global
> user id (user@foo.com) as a root, which is looked up in (what I would
> call) a root server. From it you find the directory server which stores the
> metadata for the remaining path. From this you find the data server where
> the file or data source is actually located and an ID meaningful to the
> server (like qid but can be a content sha1 sum). The directory server also
> checks if the requester is allowed access and presumably gives her a public
> key of the root user to be able to decrypt the data.
>
> Clearly, if the source is not an ordinary file, there can be no sha1 sum
> -- presumably the directory server doesn't care.
>
> The overview talks about the design being geared toward friends and family
> (ala Dropbox?) but the only thing I see that would be hard to scale is the
> fact a dir tree has an ACL. A dir server may also end up being a bottleneck.
>
> User data can be protected by the owner but the dir server needs to be
> able to read metadata such as ACL, data location etc.
>
> Not sure if the design allows for dynamic bind/mount. This would require a
> more flexible dir server structure... (I haven't read the code so this is
> pure speculation). But I'm wondering if something the CPU command can be
> implemented. May be there is a protocol to attach your own dir server.
>
> Renames are probably not handled to avoid atomicity (just speculating). Or
> may depend on a dir server.
>
> ACLs are for dir trees. From the syntax it looks like you can add more
> access in a sub tree but not remove it.
>
> I'd have preferred a capability scheme instead of ACLs -- need to think
> more about this.
>

[-- Attachment #2: Type: text/html, Size: 5577 bytes --]

Re: [9fans] Upspin - a respin of 9p?

[Aram Hăvărneanu]

On Thu, Feb 23, 2017 at 6:52 PM, Charles Forsyth
<charles.forsyth@gmail.com> wrote:
> To install this distributed file system software on my Windows machine, I
> need to install Go, which is too big for what remains on the built-in drive,
> so I pointed it at a removable one, and the installer gives the diagnostic
> "Installation directory must be on a local hard drive", which is slightly
> ironic in the circumstances.

Upspin does not currently work on Windows, but if you need Go on
Windows you don't need to use the Go installer, unzipping the zip file
will suffice (and works in any directory)

-- 
Aram Hăvărneanu

Re: [9fans] Upspin - a respin of 9p?

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 459 bytes --]

On 23 February 2017 at 17:44, Aram Hăvărneanu <aram.h@mgk.ro> wrote:

> Upspin does not currently work on Windows, but if you need Go on
> Windows you don't need to use the Go installer, unzipping the zip file
> will suffice (and works in any directory)
>

That's useful for the future, thanks. I wanted it on a big server as well
so I installed it there & copied.
What needs to be done to get upspin to work on Windows? Shallow or deep
problem?

[-- Attachment #2: Type: text/html, Size: 835 bytes --]

Re: [9fans] Upspin - a respin of 9p?

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 294 bytes --]

I see it as being orthogonal or complementary to various other existing
facilities, including 9P, ix, 9Pcloud, CliveTalk, etc, not competing or
replacing as such, although it obviously owes something to earlier
projects; and I think it can be immediately useful to me in various
contexts.

[-- Attachment #2: Type: text/html, Size: 377 bytes --]

Re: [9fans] Upspin - a respin of 9p?

[Aram Hăvărneanu]

On Thu, Feb 23, 2017 at 7:48 PM, Charles Forsyth
<charles.forsyth@gmail.com> wrote:
> What needs to be done to get upspin to work on Windows? Shallow or deep
> problem?

Well now it implements synthetic filesystems through FUSE, and FUSE is
not available on Windows. Even worse, there's no good way of
implementing synthetic filesystems in userspace on Windows.

In principle you can use upspin without kernel support for mounting
synthetic filesystems, just like you can conceivably use 9P without
the ability to mount filesystems in the namespace, but the effect of
this is unclear.

I don't have a good understanding if the usual, desired mode of
operation is through the file system or through the protocols
directly, via libraries and commands.

-- 
Aram Hăvărneanu

Re: [9fans] Upspin - a respin of 9p?

[Skip Tavakkolian]

[-- Attachment #1: Type: text/plain, Size: 989 bytes --]

hard, but not impossible; Dokan:

https://github.com/dokan-dev/dokany



On Fri, Feb 24, 2017 at 9:44 AM Aram Hăvărneanu <aram.h@mgk.ro> wrote:

> On Thu, Feb 23, 2017 at 7:48 PM, Charles Forsyth
> <charles.forsyth@gmail.com> wrote:
> > What needs to be done to get upspin to work on Windows? Shallow or deep
> > problem?
>
> Well now it implements synthetic filesystems through FUSE, and FUSE is
> not available on Windows. Even worse, there's no good way of
> implementing synthetic filesystems in userspace on Windows.
>
> In principle you can use upspin without kernel support for mounting
> synthetic filesystems, just like you can conceivably use 9P without
> the ability to mount filesystems in the namespace, but the effect of
> this is unclear.
>
> I don't have a good understanding if the usual, desired mode of
> operation is through the file system or through the protocols
> directly, via libraries and commands.
>
> --
> Aram Hăvărneanu
>
>

[-- Attachment #2: Type: text/html, Size: 1860 bytes --]