XtreemFS - a cloud file system - FAQ freqeuently asked questions

Frequently Asked Questions

General

Is XtreemFS open source?

Yes. Starting with release 1.3 all components of XtreemFS are licensed under the new BSD license. Previous releases are under GPL v2.
Who is behind XtreemFS?

XtreemFS is developed as part of the EU funded research project Contrail. For a list of developers see the team.
In which way does XtreemFS differ from network file systems, like NFS or file systems made accessible via Samba/CIFS?

Unlike common network file systems, XtreemFS is a truly distributed file system that runs across multiple servers. It can be easily scaled by adding new machines. By means of striping, it can access files with the aggregated bandwidth of multiple storage servers. Besides, it offers X.509-based authentication and encryption support, as well as features like off-site replication, which renders XtreemFS particularly useful for wide-area networks.
In which way does XtreemFS differ from Lustre?

While Lustre was made for the use in a local cluster, XtreemFS has been especially designed for wide area networks. It can be mounted and accessed from anywhere. Features like replication, RAID and checksums guarantee a high data safety and availability, and make XtreemFS less susceptible to downtimes and crashes of single servers.
I have more questions about XtreemFS. Where can I find the answers?

Please send your questions, comments and suggestions to our public mailing list. It is open to everyone, users and developers alike.

Technical

What is FUSE?

FUSE ("Filesystem in Userspace") is a Linux kernel module that simplifies the development of new file systems. It is part of most recent Linux distributions.
What is a DIR, MRC, OSD?
- DIR - Directory service. Used by the MRC, OSD and client to find other servers, resolve UUID mappings and volume names.
- MRC - Metadata and Replica catalog. Stores the volumes with their directory tree and all metadata for files in the volumes. The MRC also stores the location of the replicas of all files.
- OSD - Objedct Sorage Device. This is a regular server, not a special device. These servers store the file content and implement the replication.
What is BabuDB and why does the MRC use it?

BabuDB is a lightweight database for persistently storing large numbers of key-value pairs. It was developed to offer an efficient and optimized database backend for the MRC. Similar to Google's BigTable, it is based on LSM-trees. BabuDB is a stand-alone library distributed under the BSD license. For more information, visit our BabuDB site at Google Code.
What are the scalability limits of XtreemFS?

Currently, a single volume may contain up to tens of millions of files. So far, no limits are known to the number of volumes and services.
What is FQDN/hostname based replica selection?

By default the client will use a random replica when multiple copies of a file are available. If you use FQDN (hostname) matching for replica selection, the client will use the storage server which has the longest match compared to it`s own hostname. E.g. a client machineY.departmentX.comanyZ.com will prefer to use osd1.departmentX.companyZ.com rather than osd1.departmentY.companyZ.com.
This selection mechanism is particularly useful in static environments where you know the network configuration and geographic distribution.
How do I put a volume in the fstab?

Add the following line to your fstab:

dir.server.com/volume_name /mnt/point xtreemfs defaults,allow_other 0 0
Where does XtreemFS store its data?

MRC and DIR store all (meta)data in BabuDB, an embedded database optimized for file system metadata storage. The local directory on the MRC/DIR host in which the data resides is defined through the babudb.baseDir and babudb.logDir properties in the MRC/DIR configuration file, which point to /var/lib/xtreemfs/{mrc|dir}/database and /var/lib/xtreemfs/{mrc|dir}/db-log by default.

The OSD stores its data in the form of files and directories on the OSD's local file system. All objects reside in the object_dir directory specified in the OSD configuration file, which points to /var/lib/xtreemfs/objs by default.

Configuration

How can I configure XtreemFS services?

For each service, there is a config file in /etc/xos/xtreemfs/.
What should I do if my OSD/MRC doesn't get its own IP/hostname right?

Detecting the local hostname and public IP address does not always work. The best way is to set the public hostname or IP address of your MRCs and OSDs in the config file. Set hostname = my.host.name or hostname = 192.168.1.1
What is .autodiscover?

setting dir.host = .autodiscover in osdconfig.properties or mrcconfig.properties allows the OSD and MRC to automatically discover your directory service DIR on the local lan.

Troubleshooting

Where can I find information on known issues?

Our issue tracker contains a ticket for every known issue.
One or more XtreemFS services failed to start up.

Please look into the log files to get an idea what was going wrong. They are located in /var/log/xtreemfs.
I cannot mount XtreemFS from a remote machine.

Please check your firewall settings on the server side. Are all ports accessible? The default ports are 32636 (MRC), 32638 (DIR), and 32640 (OSD).

Please make sure that you use the DIR service hostname for mounting and not the MRC hostname!
I cannot access a mounted volume.

Please make sure that you have sufficient access rights to the volume root. Superusers and volume owners can change these rights via chmod <mode> <mountpoint>. If you try to access a mount point that belongs to a different user, please make sure that the volume is mounted with mount.xtreemfs -o allow_other.
An I/O error occurs when trying to create/access/delete a file.

The easiest way to find out what went wrong is to check the error list of the client. Run
xtfsutil --errors <mount point>
to view the list of recent errors. In addition, you can check the error logs of the services. They are located in /var/log/xtreemfs.
I cannot create new files, the client log says "... no suitable OSDs found..."
There are two ways to check the list of suitable OSDs for a volume:
- Open the MRC status page. It can be accessed via http://<MRC-host>:30636 in the default case. For each volume, a list of suitable OSDs is shown there.
- Execute xtfsutil <mountpoint> and have a look at the row "Selectable OSDs".
There may be different reasons for missing suitable OSDs:
- One or more OSDs failed to start up. Please check the log files and status pages of all OSDs to ensure that they are running.
- One or more OSDs failed to register or regularly report activity at the DIR. Please check the DIR status page to ensure that all OSDs are registered and active.
- There are no more OSDs with a sufficient amount of free disk space. Please check the OSD status page to obtain information about free disk space.
Please check whether all OSDs assigned to the file are running and reachable. This can be done as follows:
1. Get the list of all OSDs for the file: xtfsutil <file>.
2. Check whether the OSDs in (one of) all replicas in the list are running and reachable, e.g. by opening the status pages or via telnet <host> <port>.
The servers don't compile, javac complains about "unmappable character for encoding...".
Make sure your locale uses UTF-8 or set the locale for compiling:
```
> export LANG=en_US.utf8
> make server
```
.
A java.lang.UnsupportedClassVersionError is thrown when I try to start up an XtreemFS service.

Please check if your Java version is up-to-date. To run XtreemFS services, Java 1.6 or later is required. Execute echo $JAVA_HOME to find out if JAVA_HOME points to Java 1.6 installation. If no JAVA_HOME is set, execute /usr/bin/java -version to determine your default Java version.
I see the message 'buffer was finalized but not freed before!' in my server logs. What does this mean?

This message is mainly for debugging purposes. Unless it comes in great numbers, it does not indicate a serious problem. The reason is that the server failed to return an internally used buffer to a buffer pool, which, however, is done automatically when the message is shown.

Updating

update from version 1.2.x to 1.3.x

see "update from version 1.1.x to 1.2.x"
update from version 1.1.x to 1.2.x

Nothing to do here, it's all automatic :). To be on the safe side, you should always dump your MRC before updating.
update from version 1.0.0 to 1.1.x
Version 1.1.x uses a new protocol version and a different on-disk database format for the directory service (DIR) and metadata server (MRC). You can safely delete the DIR database, all servers will re-register on start up. The DIR database is ususally stored in /var/lib/xtreemfs/dir/.
The MRC database must be dumped before installing the new release!
1. use xtfs_mrcdbtool -mrc oncrpc://mrchost:32636 dump /path/to/dumpfile to save the database to an XML file.
2. check that the file exists on the MRC host and is not empty!
3. shut down the MRC /etc/init.d/xtreemfs-mrc stop (and the DIR and OSD if they are still running)
4. remove the MRC database /var/lib/xtreemfs/mrc/
5. install the 1.1 release of XtreemFS
6. start the MRC /etc/init.d/xtreemfs-mrc start
7. use xtfs_mrcdbtool -mrc oncrpc://mrchost:32636 restore /path/to/dumpfile to restore the database from the XML file.