From: Steven McDonald Date: Sat, 19 Jan 2013 17:28:15 +0000 (+1100) Subject: Trove hackfest work with appropriate copyright notices and licence files added X-Git-Tag: hackfest-20130118 X-Git-Url: http://git.steven-mcdonald.id.au/?p=hurd%2Ftrovefs.git;a=commitdiff_plain;h=59c2da494dd311df4dc61506a1eda6918e7526e5 Trove hackfest work with appropriate copyright notices and licence files added --- 59c2da494dd311df4dc61506a1eda6918e7526e5 diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..a4603f7 --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +trovefs diff --git a/COPYING.GPL b/COPYING.GPL new file mode 100644 index 0000000..d159169 --- /dev/null +++ b/COPYING.GPL @@ -0,0 +1,339 @@ + GNU GENERAL PUBLIC LICENSE + Version 2, June 1991 + + Copyright (C) 1989, 1991 Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +License is intended to guarantee your freedom to share and change free +software--to make sure the software is free for all its users. This +General Public License applies to most of the Free Software +Foundation's software and to any other program whose authors commit to +using it. (Some other Free Software Foundation software is covered by +the GNU Lesser General Public License instead.) You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +this service if you wish), that you receive source code or can get it +if you want it, that you can change the software or use pieces of it +in new free programs; and that you know you can do these things. + + To protect your rights, we need to make restrictions that forbid +anyone to deny you these rights or to ask you to surrender the rights. +These restrictions translate to certain responsibilities for you if you +distribute copies of the software, or if you modify it. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must give the recipients all the rights that +you have. You must make sure that they, too, receive or can get the +source code. And you must show them these terms so they know their +rights. + + We protect your rights with two steps: (1) copyright the software, and +(2) offer you this license which gives you legal permission to copy, +distribute and/or modify the software. + + Also, for each author's protection and ours, we want to make certain +that everyone understands that there is no warranty for this free +software. If the software is modified by someone else and passed on, we +want its recipients to know that what they have is not the original, so +that any problems introduced by others will not reflect on the original +authors' reputations. + + Finally, any free program is threatened constantly by software +patents. We wish to avoid the danger that redistributors of a free +program will individually obtain patent licenses, in effect making the +program proprietary. To prevent this, we have made it clear that any +patent must be licensed for everyone's free use or not licensed at all. + + The precise terms and conditions for copying, distribution and +modification follow. + + GNU GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License applies to any program or other work which contains +a notice placed by the copyright holder saying it may be distributed +under the terms of this General Public License. The "Program", below, +refers to any such program or work, and a "work based on the Program" +means either the Program or any derivative work under copyright law: +that is to say, a work containing the Program or a portion of it, +either verbatim or with modifications and/or translated into another +language. (Hereinafter, translation is included without limitation in +the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running the Program is not restricted, and the output from the Program +is covered only if its contents constitute a work based on the +Program (independent of having been made by running the Program). +Whether that is true depends on what the Program does. + + 1. You may copy and distribute verbatim copies of the Program's +source code as you receive it, in any medium, provided that you +conspicuously and appropriately publish on each copy an appropriate +copyright notice and disclaimer of warranty; keep intact all the +notices that refer to this License and to the absence of any warranty; +and give any other recipients of the Program a copy of this License +along with the Program. + +You may charge a fee for the physical act of transferring a copy, and +you may at your option offer warranty protection in exchange for a fee. + + 2. You may modify your copy or copies of the Program or any portion +of it, thus forming a work based on the Program, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) You must cause the modified files to carry prominent notices + stating that you changed the files and the date of any change. + + b) You must cause any work that you distribute or publish, that in + whole or in part contains or is derived from the Program or any + part thereof, to be licensed as a whole at no charge to all third + parties under the terms of this License. + + c) If the modified program normally reads commands interactively + when run, you must cause it, when started running for such + interactive use in the most ordinary way, to print or display an + announcement including an appropriate copyright notice and a + notice that there is no warranty (or else, saying that you provide + a warranty) and that users may redistribute the program under + these conditions, and telling the user how to view a copy of this + License. (Exception: if the Program itself is interactive but + does not normally print such an announcement, your work based on + the Program is not required to print an announcement.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Program, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Program, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program +with the Program (or with a work based on the Program) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may copy and distribute the Program (or a work based on it, +under Section 2) in object code or executable form under the terms of +Sections 1 and 2 above provided that you also do one of the following: + + a) Accompany it with the complete corresponding machine-readable + source code, which must be distributed under the terms of Sections + 1 and 2 above on a medium customarily used for software interchange; or, + + b) Accompany it with a written offer, valid for at least three + years, to give any third party, for a charge no more than your + cost of physically performing source distribution, a complete + machine-readable copy of the corresponding source code, to be + distributed under the terms of Sections 1 and 2 above on a medium + customarily used for software interchange; or, + + c) Accompany it with the information you received as to the offer + to distribute corresponding source code. (This alternative is + allowed only for noncommercial distribution and only if you + received the program in object code or executable form with such + an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for +making modifications to it. For an executable work, complete source +code means all the source code for all modules it contains, plus any +associated interface definition files, plus the scripts used to +control compilation and installation of the executable. However, as a +special exception, the source code distributed need not include +anything that is normally distributed (in either source or binary +form) with the major components (compiler, kernel, and so on) of the +operating system on which the executable runs, unless that component +itself accompanies the executable. + +If distribution of executable or object code is made by offering +access to copy from a designated place, then offering equivalent +access to copy the source code from the same place counts as +distribution of the source code, even though third parties are not +compelled to copy the source along with the object code. + + 4. You may not copy, modify, sublicense, or distribute the Program +except as expressly provided under this License. Any attempt +otherwise to copy, modify, sublicense or distribute the Program is +void, and will automatically terminate your rights under this License. +However, parties who have received copies, or rights, from you under +this License will not have their licenses terminated so long as such +parties remain in full compliance. + + 5. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Program or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Program (or any work based on the +Program), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Program or works based on it. + + 6. Each time you redistribute the Program (or any work based on the +Program), the recipient automatically receives a license from the +original licensor to copy, distribute or modify the Program subject to +these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties to +this License. + + 7. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Program at all. For example, if a patent +license would not permit royalty-free redistribution of the Program by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Program. + +If any portion of this section is held invalid or unenforceable under +any particular circumstance, the balance of the section is intended to +apply and the section as a whole is intended to apply in other +circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system, which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 8. If the distribution and/or use of the Program is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Program under this License +may add an explicit geographical distribution limitation excluding +those countries, so that distribution is permitted only in or among +countries not thus excluded. In such case, this License incorporates +the limitation as if written in the body of this License. + + 9. The Free Software Foundation may publish revised and/or new versions +of the General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies a version number of this License which applies to it and "any +later version", you have the option of following the terms and conditions +either of that version or of any later version published by the Free +Software Foundation. If the Program does not specify a version number of +this License, you may choose any version ever published by the Free Software +Foundation. + + 10. If you wish to incorporate parts of the Program into other free +programs whose distribution conditions are different, write to the author +to ask for permission. For software which is copyrighted by the Free +Software Foundation, write to the Free Software Foundation; we sometimes +make exceptions for this. Our decision will be guided by the two goals +of preserving the free status of all derivatives of our free software and +of promoting the sharing and reuse of software generally. + + NO WARRANTY + + 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY +FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN +OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES +PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED +OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS +TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE +PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, +REPAIR OR CORRECTION. + + 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR +REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING +OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED +TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY +YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER +PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGES. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +convey the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License along + with this program; if not, write to the Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +Also add information on how to contact you by electronic and paper mail. + +If the program is interactive, make it output a short notice like this +when it starts in an interactive mode: + + Gnomovision version 69, Copyright (C) year name of author + Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, the commands you use may +be called something other than `show w' and `show c'; they could even be +mouse-clicks or menu items--whatever suits your program. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the program + `Gnomovision' (which makes passes at compilers) written by James Hacker. + + , 1 April 1989 + Ty Coon, President of Vice + +This General Public License does not permit incorporating your program into +proprietary programs. If your program is a subroutine library, you may +consider it more useful to permit linking proprietary applications with the +library. If this is what you want to do, use the GNU Lesser General +Public License instead of this License. diff --git a/COPYING.WTFPL b/COPYING.WTFPL new file mode 100644 index 0000000..ee7d6a5 --- /dev/null +++ b/COPYING.WTFPL @@ -0,0 +1,14 @@ + DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE + Version 2, December 2004 + + Copyright (C) 2004 Sam Hocevar + + Everyone is permitted to copy and distribute verbatim or modified + copies of this license document, and changing it is allowed as long + as the name is changed. + + DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. You just DO WHAT THE FUCK YOU WANT TO. + diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..31273a9 --- /dev/null +++ b/Makefile @@ -0,0 +1,14 @@ +# Copyright (C) 2013 Steven McDonald +# +# This program is free software. It comes without any warranty, to the +# extent permitted by applicable law. You can redistribute it and/or +# modify it under the terms of the Do What The Fuck You Want To Public +# License, Version 2, as published by Sam Hocevar. See the file +# COPYING.WTFPL accompanying this distribution or http://www.wtfpl.net/ +# for more details. + +all: + gcc --std=c99 -D_GNU_SOURCE -g -Wall -lihash -lnetfs -ls3 -o trovefs dir.c fs.c netfs.c node.c s3.c trovefs.c + +readdir: + gcc --std=c99 -D_GNU_SOURCE -g -Wall -o readdir readdir.c diff --git a/dir.c b/dir.c new file mode 100644 index 0000000..41a6d08 --- /dev/null +++ b/dir.c @@ -0,0 +1,244 @@ +/* + * Copyright (C) 1997,98,2002 Free Software Foundation, Inc. + * Copyright (C) 2013 Steven McDonald + * + * ftpfs translator written by Miles Bader + * Modified for trovefs by Steven McDonald + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2, or (at your option) + * any later version. + * + * This software is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111, USA. + */ + +#include +#include + +#include "trovefs.h" + +#define INIT_HTABLE_LEN 5 + +struct dir_fetch_state +{ + struct trovefs_dir *dir; + struct trovefs_dir_entry **prev_entry_next_p; +}; + +static void +insert (struct trovefs_dir_entry *e, struct trovefs_dir_entry **htable, size_t htable_len) +{ + struct trovefs_dir_entry **t = &htable[e->hv % htable_len]; + if (*t) + (*t)->self_p = &e->next; + e->next = *t; + e->self_p = t; + *t = e; +} + +static error_t +rehash (struct trovefs_dir *dir, size_t new_len) +{ + int i; + size_t old_len = dir->htable_len; + struct trovefs_dir_entry **old_htable = dir->htable; + struct trovefs_dir_entry **new_htable = malloc (new_len * sizeof (struct trovefs_dir_entry *)); + + if (! new_htable) + return ENOMEM; + + bzero (new_htable, new_len * sizeof (struct trovefs_dir_entry *)); + + for (i = 0; i < old_len; i++) + while (old_htable[i]) + { + struct trovefs_dir_entry *e = old_htable[i]; + old_htable[i] = e->next; + insert (e, new_htable, new_len); + } + + free (old_htable); + + dir->htable = new_htable; + dir->htable_len = new_len; + + return 0; +} + +static size_t +hash (const char *name) +{ + size_t hv = 0; + while (*name) + hv = ((hv << 5) + *name++) & 0xFFFFFF; + return hv; +} + +struct trovefs_dir_entry * +lookup (struct trovefs_dir *dir, const char *name, int add) +{ + size_t hv = hash (name); + struct trovefs_dir_entry *h = dir->htable[hv % dir->htable_len], *e = h; + + while (e && strcmp (name, e->name) != 0) + e = e->next; + + if (!e && add) + { + if (dir->num_entries > dir->htable_len) + if (rehash(dir, (dir->htable_len + 1) * 2 - 1) != 0) + return 0; + + e = malloc (sizeof *e); + if (e) + { + e->hv = hv; + e->name = strdup (name); + e->node = 0; + e->dir = dir; + e->stat_timestamp = 0; + bzero (&e->stat, sizeof (e->stat)); + e->noent = 0; + e->valid = 0; + e->name_timestamp = e->stat_timestamp = 0; + e->ordered_next = 0; + e->ordered_self_p = 0; + e->next = 0; + e->self_p = 0; + insert (e, dir->htable, dir->htable_len); + dir->num_entries++; + } + } + + return e; +} + +error_t +trovefs_dir_create (struct trovefs *fs, struct node *node, const char *rmt_path, struct trovefs_dir **dir) +{ + struct trovefs_dir *new = malloc (sizeof (struct trovefs_dir)); + struct trovefs_dir_entry **htable = calloc (INIT_HTABLE_LEN, sizeof (struct trovefs_dir_entry *)); + + if (!new || !htable) + { + if (new) + free (new); + if (htable) + free (htable); + return ENOMEM; + } + + pthread_spin_lock (&netfs_node_refcnt_lock); + node->references++; + pthread_spin_unlock (&netfs_node_refcnt_lock); + + new->num_entries = 0; + new->num_live_entries = 0; + new->htable_len = INIT_HTABLE_LEN; + new->htable = htable; + new->ordered = 0; + new->rmt_path = rmt_path; + new->fs = fs; + new->node = node; + new->stat_timestamp = 0; + new->name_timestamp = 0; + new->bulk_stat_base_stamp = 0; + new->bulk_stat_count_first_half = 0; + new->bulk_stat_count_second_half = 0; + + *dir = new; + + return 0; +} + +error_t +trovefs_dir_null_lookup (struct trovefs_dir *dir, struct node **node) +{ + struct trovefs_dir_entry *e; + error_t err = 0; + + e = lookup (dir, "", 1); + if (! e) + return ENOMEM; + + if (! e->noent) + { + pthread_spin_lock (&netfs_node_refcnt_lock); + if (e->node) + e->node->references++; + pthread_spin_unlock (&netfs_node_refcnt_lock); + + if (! e->node) + { + err = trovefs_create_node (e, dir->rmt_path, &e->node); + + if (!err && dir->num_live_entries++ == 0) + { + pthread_spin_lock (&netfs_node_refcnt_lock); + dir->node->references++; + pthread_spin_unlock (&netfs_node_refcnt_lock); + } + } + + if (! err) + *node = e->node; + } + else + err = ENOENT; + + return err; +} + +error_t +trovefs_refresh_node (struct node *node) +{ + error_t err = 0; + struct netnode *nn = node->nn; + struct trovefs_dir_entry *entry = nn->dir_entry; + + if (! entry) + return 0; + else + { + pthread_mutex_lock (&entry->dir->node->lock); + + if (! entry->self_p) + { + nn->dir_entry = 0; + free_entry (entry); + pthread_mutex_unlock (&entry->dir->node->lock); + return 0; + } + else + { + err = trovefs_s3_get_names (entry->dir); + if (!err && entry->noent) + err = ENOENT; + } + + if (err == ENOENT) + { + entry->noent = 1; + } + + pthread_mutex_unlock (&entry->dir->node->lock); + } + + return err; +} + +void +free_entry (struct trovefs_dir_entry *e) +{ + assert (! e->self_p); + free (e->name); + free (e); +} diff --git a/fs.c b/fs.c new file mode 100644 index 0000000..68a4f89 --- /dev/null +++ b/fs.c @@ -0,0 +1,87 @@ +/* + * Copyright (C) 1997, 2001, 2003 Free Software Foundation, Inc. + * Copyright (C) 2013 Steven McDonald + * + * ftpfs translator written by Miles Bader + * Modified for trovefs by Steven McDonald + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2, or (at your option) + * any later version. + * + * This software is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111, USA. + * + */ + +#include +#include + +#include "trovefs.h" + +error_t +trovefs_create (char *hostname, char *bucket, char *access_key_id, char *secret_access_key, int fsid, struct trovefs **fs) +{ + error_t err; + S3Status s3err; + struct trovefs_dir *super_root_dir; + struct node *super_root; + struct trovefs *new = malloc (sizeof (struct trovefs)); + + if (! new) + return ENOMEM; + + new->fsid = fsid; + new->next_inode = 2; + + s3err = S3_initialize (TROVEFS_USER_AGENT, S3_INIT_ALL, hostname); + switch (s3err) + { + case S3StatusOK: + break; + case S3StatusUriTooLong: + return EINVAL; + case S3StatusInternalError: + return EGRATUITOUS; + case S3StatusOutOfMemory: + return ENOMEM; + default: + return EGRATUITOUS; + } + + new->s3_ctx = malloc (sizeof (S3BucketContext)); + new->s3_ctx->hostName = hostname; + new->s3_ctx->bucketName = bucket; + new->s3_ctx->protocol = S3ProtocolHTTPS; + new->s3_ctx->uriStyle = S3UriStyleVirtualHost; + new->s3_ctx->accessKeyId = access_key_id; + new->s3_ctx->secretAccessKey = secret_access_key; + + hurd_ihash_init (&new->inode_mappings, offsetof (struct trovefs_dir_entry, inode_locp)); + pthread_spin_init (&new->inode_mappings_lock, PTHREAD_PROCESS_PRIVATE); + + super_root = netfs_make_node (0); + if (! super_root) + err = ENOMEM; + else { + err = trovefs_dir_create (new, super_root, "/", &super_root_dir); + if (! err) + err = trovefs_dir_null_lookup (super_root_dir, &new->root); + } + + if (err) { + hurd_ihash_destroy (&new->inode_mappings); + free (new); + } + else + *fs = new; + + return err; +} diff --git a/netfs.c b/netfs.c new file mode 100644 index 0000000..f9f7d2f --- /dev/null +++ b/netfs.c @@ -0,0 +1,304 @@ +/* + * Copyright (C) 1997, 1998, 1999, 2001, 2007 Free Software Foundation, Inc. + * Copyright (C) 2013 Steven McDonald + * + * ftpfs translator written by Miles Bader + * Modified for trovefs by Steven McDonald + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2, or (at your option) + * any later version. + * + * This software is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111, USA. + */ + +#include +#include +#include +#include +#include + +#include "trovefs.h" + +error_t +netfs_attempt_link (struct iouser *user, struct node *dir, struct node *file, char *name, int excl) +{ + fprintf (stderr, "a\n"); + return EOPNOTSUPP; +} + +error_t +netfs_attempt_create_file (struct iouser *user, struct node *dir, char *name, mode_t mode, struct node **node) +{ + fprintf (stderr, "b\n"); + return EOPNOTSUPP; +} + +error_t +netfs_attempt_chown (struct iouser *cred, struct node *node, uid_t uid, uid_t gid) +{ + fprintf (stderr, "c\n"); + return EOPNOTSUPP; +} + +void +netfs_node_norefs (struct node *node) +{ + fprintf (stderr, "d\n"); + return; +} + +error_t +netfs_attempt_mksymlink (struct iouser *cred, struct node *node, char *name) +{ + fprintf (stderr, "e\n"); + return EOPNOTSUPP; +} + +error_t +netfs_attempt_unlink (struct iouser *user, struct node *dir, char *name) +{ + fprintf (stderr, "f\n"); + return EOPNOTSUPP; +} + +error_t +netfs_attempt_syncfs (struct iouser *cred, int wait) +{ + return 0; +} + +error_t +netfs_attempt_lookup (struct iouser *user, struct node *dir, char *name, struct node **node) +{ + fprintf (stderr, "h\n"); + return EOPNOTSUPP; +} + +error_t +netfs_attempt_chauthor (struct iouser *cred, struct node *node, uid_t author) +{ + fprintf (stderr, "i\n"); + return EOPNOTSUPP; +} + +error_t +netfs_attempt_mkdir (struct iouser *user, struct node *dir, char *name, mode_t mode) +{ + fprintf (stderr, "j\n"); + return EOPNOTSUPP; +} + +error_t +netfs_attempt_mkfile (struct iouser *user, struct node *dir, mode_t mode, struct node **node) +{ + fprintf (stderr, "k\n"); + return EOPNOTSUPP; +} + +error_t +netfs_attempt_readlink (struct iouser *user, struct node *node, char *buf) +{ + fprintf (stderr, "l\n"); + return EOPNOTSUPP; +} + +error_t +netfs_attempt_sync (struct iouser *cred, struct node *node, int wait) +{ + fprintf (stderr, "m\n"); + return EOPNOTSUPP; +} + +error_t +netfs_attempt_statfs (struct iouser *cred, struct node *node, fsys_statfsbuf_t *st) +{ + fprintf (stderr, "n\n"); + return EOPNOTSUPP; +} + +error_t +netfs_validate_stat (struct node *node, struct iouser *cred) +{ + return trovefs_refresh_node (node); +} + +error_t +netfs_attempt_chmod (struct iouser *cred, struct node *node, mode_t mode) +{ + fprintf (stderr, "p\n"); + return EOPNOTSUPP; +} + +error_t +netfs_attempt_set_size (struct iouser *cred, struct node *node, loff_t size) +{ + fprintf (stderr, "q\n"); + return EOPNOTSUPP; +} + +error_t +netfs_attempt_rmdir (struct iouser *user, struct node *dir, char *name) +{ + fprintf (stderr, "r\n"); + return EOPNOTSUPP; +} + +#define DIRENTS_CHUNK_SIZE (8*1024) +#define DIRENT_ALIGN 4 +#define DIRENT_NAME_OFFS offsetof (struct dirent, d_name) +#define DIRENT_LEN(name_len) ((DIRENT_NAME_OFFS + (name_len) + 1 + (DIRENT_ALIGN - 1)) & ~(DIRENT_ALIGN - 1)) +error_t +netfs_get_dirents (struct iouser *cred, struct node *dir, int first_entry, int max_entries, char **data, mach_msg_type_number_t *data_len, vm_size_t max_data_len, int *data_entries) +{ + error_t err = trovefs_refresh_node (dir); + + if (err) + return err; + + struct trovefs_dir *realdir = dir->nn->dir_entry->dir; + + if (! realdir) + return ENOTDIR; + + struct trovefs_dir_entry *e; + e = realdir->ordered; + + while (first_entry-- > 0) + if (! e) + { + max_entries = 0; + break; + } + else + e = e->ordered_next; + + if (max_entries != 0) + { + size_t size = (max_data_len == 0 || max_data_len > DIRENTS_CHUNK_SIZE ? DIRENTS_CHUNK_SIZE : max_data_len); + *data = mmap (0, size, PROT_READ|PROT_WRITE, MAP_ANON, 0, 0); + err = ((void *) *data == (void *) -1) ? errno : 0; + + if (! err) + { + char *p = *data; + int count = 0; + + while ((max_entries == -1 || count < max_entries) && e) + { + struct dirent hdr; + size_t name_len = strlen (e->name); + size_t sz = DIRENT_LEN (name_len); + int entry_type = e->stat_timestamp ? IFTODT (e->stat.st_mode) : DT_UNKNOWN; + + if ((p - *data) + sz > size) + { + if (max_data_len > 0) + break; + else + { + vm_address_t extension = (vm_address_t)(*data + size); + err = vm_allocate (mach_task_self (), &extension, DIRENTS_CHUNK_SIZE, 0); + if (err) + break; + size += DIRENTS_CHUNK_SIZE; + } + } + + hdr.d_namlen = name_len; + hdr.d_fileno = e->stat.st_ino; + hdr.d_reclen = sz; + hdr.d_type = entry_type; + + memcpy (p, &hdr, DIRENT_NAME_OFFS); + strcpy (p + DIRENT_NAME_OFFS, e->name); + p += sz; + + count++; + e = e->ordered_next; + } + + if (err) + munmap (*data, size); + else + { + vm_address_t alloc_end = (vm_address_t)(*data + size); + vm_address_t real_end = round_page (p); + if (alloc_end > real_end) + munmap ((caddr_t) real_end, alloc_end - real_end); + *data_len = p - *data; + *data_entries = count; + } + } + } + else + { + *data_len = 0; + *data_entries = 0; + } + + return err; +} + +error_t +netfs_attempt_chflags (struct iouser *cred, struct node *node, int flags) +{ + fprintf (stderr, "t\n"); + return EOPNOTSUPP; +} + +error_t +netfs_attempt_write (struct iouser *cred, struct node *node, loff_t offset, size_t *len, void *data) +{ + fprintf (stderr, "u\n"); + return EOPNOTSUPP; +} + +error_t +netfs_attempt_read (struct iouser *cred, struct node *node, loff_t offset, size_t *len, void *data) +{ + fprintf (stderr, "v\n"); + return EOPNOTSUPP; +} + +error_t +netfs_report_access (struct iouser *cred, struct node *node, int *types) +{ + fprintf (stderr, "w\n"); + return EOPNOTSUPP; +} + +error_t +netfs_attempt_mkdev (struct iouser *cred, struct node *node, mode_t type, dev_t indexes) +{ + fprintf (stderr, "x\n"); + return EOPNOTSUPP; +} + +error_t +netfs_attempt_utimes (struct iouser *cred, struct node *node, struct timespec *atime, struct timespec *mtime) +{ + fprintf (stderr, "y\n"); + return EOPNOTSUPP; +} + +error_t +netfs_check_open_permissions (struct iouser *user, struct node *node, int flags, int newnode) +{ + return 0; +} + +error_t +netfs_attempt_rename (struct iouser *user, struct node *fromdir, char *fromname, struct node *todir, char *toname, int excl) +{ + fprintf (stderr, "=\n"); + return EOPNOTSUPP; +} diff --git a/node.c b/node.c new file mode 100644 index 0000000..5988aaf --- /dev/null +++ b/node.c @@ -0,0 +1,67 @@ +/* + * Copyright (C) 1997, 2006 Free Software Foundation, Inc. + * Copyright (C) 2013 Steven McDonald + * + * ftpfs translator written by Miles Bader + * Modified for trovefs by Steven McDonald + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2, or (at your option) + * any later version. + * + * This software is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111, USA. + */ + +#include +#include +#include +#include "trovefs.h" + +error_t +trovefs_create_node (struct trovefs_dir_entry *e, const char *rmt_path, struct node **node) +{ + struct node *new; + struct netnode *nn = malloc (sizeof (struct netnode)); + error_t err; + + if (! nn) + return ENOMEM; + + nn->fs = e->dir->fs; + nn->dir_entry = e; + nn->contents = 0; + nn->dir = 0; + nn->rmt_path = strdup (rmt_path); + nn->ncache_next = nn->ncache_prev = 0; + + new = netfs_make_node (nn); + if (! new) + { + free (nn); + return ENOMEM; + } + + pthread_spin_lock (&nn->fs->inode_mappings_lock); + err = hurd_ihash_add (&nn->fs->inode_mappings, e->stat.st_ino, e); + pthread_spin_unlock (&nn->fs->inode_mappings_lock); + + if (err) + { + free (nn); + free (new); + return err; + } + + e->node = new; + *node = new; + + return 0; +} diff --git a/readdir.rb b/readdir.rb new file mode 100755 index 0000000..c95ca27 --- /dev/null +++ b/readdir.rb @@ -0,0 +1,14 @@ +#!/usr/bin/env ruby + +# Copyright (C) 2013 Steven McDonald +# +# This program is free software. It comes without any warranty, to the +# extent permitted by applicable law. You can redistribute it and/or +# modify it under the terms of the Do What The Fuck You Want To Public +# License, Version 2, as published by Sam Hocevar. See the file +# COPYING.WTFPL accompanying this distribution or http://www.wtfpl.net/ +# for more details. + +d = Dir.new("/home/steven/trovefs/foo") +i = nil +puts i while i = d.read diff --git a/s3.c b/s3.c new file mode 100644 index 0000000..ae5b9cb --- /dev/null +++ b/s3.c @@ -0,0 +1,129 @@ +/* + * Copyright (C) 2013 Steven McDonald + * + * This program is free software. It comes without any warranty, to the + * extent permitted by applicable law. You can redistribute it and/or + * modify it under the terms of the Do What The Fuck You Want To Public + * License, Version 2, as published by Sam Hocevar. See the file + * COPYING.WTFPL accompanying this distribution or http://www.wtfpl.net/ + * for more details. + */ + +#include +#include +#include +#include "trovefs.h" + +static char *keys[TROVEFS_MAX_LIST_KEYS]; + +typedef struct list_bucket_callback_data +{ + int isTruncated; + char nextMarker[1024]; + int keyCount; + int allDetails; +} list_bucket_callback_data; + +static S3Status +responsePropertiesCallback (const S3ResponseProperties *properties, void *callbackData) +{ + return S3StatusOK; +} + +static void +responseCompleteCallback (S3Status status, const S3ErrorDetails *error, void *callbackData) +{ + return; +} + +static S3Status +list_bucket_callback ( + int isTruncated, + const char *nextMarker, + int contentsCount, + const S3ListBucketContent *contents, + int commonPrefixesCount, + const char **commonPrefixes, + void *callbackData +) { + list_bucket_callback_data *data = (list_bucket_callback_data *) callbackData; + + data->isTruncated = isTruncated; + + if ((!nextMarker || !nextMarker[0]) && contentsCount) + { + nextMarker = contents[contentsCount - 1].key; + } + if (nextMarker) + { + snprintf (data->nextMarker, sizeof (data->nextMarker), "%s", nextMarker); + } + else + { + data->nextMarker[0] = 0; + } + + for (int i = 0; i < contentsCount; i++) + { + const S3ListBucketContent *content = &(contents[i]); + keys[i] = malloc (sizeof (content->key)); + strcpy (keys[i], content->key); + } + keys[contentsCount] = malloc (1); + keys[contentsCount][0] = '\0'; + return S3StatusOK; +} + +error_t +trovefs_s3_get_names (struct trovefs_dir *dir) +{ + error_t err = 0; + struct trovefs *fs = dir->fs; + S3BucketContext *bucket = fs->s3_ctx; + + S3ListBucketHandler list_bucket_handler = + { + { &responsePropertiesCallback, &responseCompleteCallback }, + &list_bucket_callback + }; + + list_bucket_callback_data data; + + S3_list_bucket ( + bucket, + dir->rmt_path+1, + "", + "", + TROVEFS_MAX_LIST_KEYS, + 0, + &list_bucket_handler, + &data + ); + + struct trovefs_dir_entry *e = 0; + struct trovefs_dir_entry *p; + + for (int i = 0; i < TROVEFS_MAX_LIST_KEYS; i++) + { + if (!keys[i] || keys[i][0] == '\0') + break; + p = e; + e = lookup (dir, keys[i], 1); + + if (! e) + return ENOMEM; + + e->valid = 1; + + if (! e->ordered_self_p) + { + e->ordered_self_p = p ? &p->next : &dir->ordered; + if (*e->ordered_self_p) + (*e->ordered_self_p)->ordered_self_p = &e->ordered_next; + e->ordered_next = *e->ordered_self_p; + *e->ordered_self_p = e; + } + } + + return err; +} diff --git a/trovefs.c b/trovefs.c new file mode 100644 index 0000000..8ecceb7 --- /dev/null +++ b/trovefs.c @@ -0,0 +1,134 @@ +/* + * Copyright (C) 1997,98,2002 Free Software Foundation Inc. + * Copyright (C) 2013 Steven McDonald + * + * ftpfs translator written by Miles Bader + * Modified for trovefs by Steven McDonald + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2, or (at your option) + * any later version. + * + * This software is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111, USA. + */ + +#include +#include +#include +//#include +#include +#include "trovefs.h" + +char *netfs_server_name = "trovefs"; +char *netfs_server_version = TROVEFS_VERSION; + +const char *argp_program_version = "trovefs " TROVEFS_VERSION; + +static char args_doc[] = "BUCKET"; +static char doc[] = "Trove filesystem translator." +"\vIf --hostname is not specified, it defaults to" +" \"" TROVEFS_DEFAULT_HOSTNAME "\"."; + +static struct argp_option options[] = +{ + {"hostname", 'h', "HOSTNAME", 0, "S3 server hostname to connect to"}, + {"access-key-id", 'i', "ID", 0, "Access key ID for S3 server"}, + {"secret-access-key", 'k', "KEY", 0, "Secret access key for S3 server"}, + + {0} +}; + +struct arguments +{ + char *args[1]; + char *hostname; + char *access_key_id; + char *secret_access_key; +}; + +static error_t +parse_opt (int key, char *arg, struct argp_state *state) +{ + struct arguments *arguments = state->input; + + switch (key) + { + case 'h': + arguments->hostname = arg; + break; + case 'i': + arguments->access_key_id = arg; + break; + case 'k': + arguments->secret_access_key = arg; + break; + case ARGP_KEY_ARG: + if (state->arg_num >= 1) + argp_usage (state); + arguments->args[state->arg_num] = arg; + break; + case ARGP_KEY_END: + if (state->arg_num < 1) + argp_usage (state); + break; + default: + return ARGP_ERR_UNKNOWN; + } + return 0; +} + +static struct argp argp = { options, parse_opt, args_doc, doc }; + +int netfs_maxsymlinks = 0; + +int +main (int argc, char *argv[]) +{ + error_t err; + mach_port_t bootstrap, underlying_node; + io_statbuf_t underlying_stat; + struct arguments *arguments = malloc (sizeof (struct arguments)); + arguments->hostname = TROVEFS_DEFAULT_HOSTNAME; + + argp_parse (&argp, argc, argv, 0, 0, arguments); + + struct trovefs *trovefs; + + task_get_bootstrap_port (mach_task_self (), &bootstrap); + + netfs_init (); + + err = trovefs_create ( + arguments->hostname, + arguments->args[0], // Bucket. + arguments->access_key_id, + arguments->secret_access_key, + getpid (), + &trovefs + ); + if (err) + return 1; +// FIXME: Why the fuck is 'error' broken? +// error (1, err, "%s:%s", arguments->hostname, arguments->args[0]); + + netfs_root_node = trovefs->root; + + underlying_node = netfs_startup (bootstrap, 0); + err = io_stat (underlying_node, &underlying_stat); + if (err) + return 2; + + netfs_root_node->nn_stat = underlying_stat; + netfs_root_node->nn_stat.st_mode = S_IFDIR | (underlying_stat.st_mode & ~S_IFMT & ~S_ITRANS); + + for (;;) + netfs_server_loop (); +} diff --git a/trovefs.h b/trovefs.h new file mode 100644 index 0000000..44de5a6 --- /dev/null +++ b/trovefs.h @@ -0,0 +1,144 @@ +/* + * Copyright (C) 1997 Free Software Foundation, Inc. + * Copyright (C) 2013 Steven McDonald + * + * ftpfs translator written by Miles Bader + * Modified for trovefs by Steven McDonald + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2, or (at your option) + * any later version. + * + * This software is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111, USA. + */ + +#ifndef __TROVEFS_H__ +#define __TROVEFS_H__ + +#include +#include +#include +#include + +#define TROVEFS_VERSION "0.0" + +#define TROVEFS_DEFAULT_HOSTNAME "beta.anchortrove.com" +#define TROVEFS_USER_AGENT "trovefs " TROVEFS_VERSION " (GNU)" +#define TROVEFS_REMOTE_PATH_MALLOC_SIZE 128 +#define TROVEFS_MAX_LIST_KEYS 10000 + +struct trovefs_dir_entry +{ + char *name; + size_t hv; + + struct node *node; + + struct stat stat; + time_t stat_timestamp; + + struct trovefs_dir *dir; + + struct trovefs_dir_entry *next, **self_p; + + struct trovefs_dir_entry *ordered_next, **ordered_self_p; + + time_t name_timestamp; + + hurd_ihash_locp_t inode_locp; + + int noent : 1; + int valid : 1; +}; + +struct trovefs_dir +{ + size_t num_entries; + + size_t num_live_entries; + + struct trovefs_dir_entry **htable; + size_t htable_len; + + struct trovefs_dir_entry *ordered; + + struct node *node; + + struct trovefs *fs; + + const char *rmt_path; + + time_t stat_timestamp; + time_t name_timestamp; + + time_t bulk_stat_base_stamp; + unsigned bulk_stat_count_first_half; + unsigned bulk_stat_count_second_half; +}; + +struct netnode +{ + struct trovefs *fs; + + struct trovefs_dir_entry *dir_entry; + + const char *rmt_path; + + struct ccache *contents; + + struct trovefs_dir *dir; + + struct node *ncache_next, *ncache_prev; +}; + +struct trovefs +{ + /* Filesystem root. */ + struct node *root; + + /* Context for S3 requests. */ + S3BucketContext *s3_ctx; + + ino_t next_inode; + int fsid; + + struct hurd_ihash inode_mappings; + pthread_spinlock_t inode_mappings_lock; +}; + +error_t +trovefs_create (char *hostname, char *bucket, char *access_key_id, char *secret_access_key, int fsid, struct trovefs **fs); + +error_t +trovefs_dir_create (struct trovefs *fs, struct node *node, const char *rmt_path, struct trovefs_dir **dir); + +error_t +trovefs_dir_null_lookup (struct trovefs_dir *dir, struct node **node); + +error_t +trovefs_create_node (struct trovefs_dir_entry *e, const char *rmt_path, struct node **node); + +error_t +trovefs_refresh_node (struct node *node); + +error_t +refresh_dir (struct trovefs_dir *dir, int update_stats, struct trovefs_dir_entry *preserve_entry); + +void +free_entry (struct trovefs_dir_entry *e); + +error_t +trovefs_s3_get_names (struct trovefs_dir *dir); + +struct trovefs_dir_entry * +lookup (struct trovefs_dir *dir, const char *name, int add); + +#endif // __TROVEFS_H__