This is particularly important for node failure and recovery, but can also be used in general to provide a more flexible environment. The management and configuration of the entire cluster as a unit should be possible, as opposed to configuring each of the cluster nodes individually. The file system image the cluster will provide will be made of a single tree and will be homogeneous regardless of the cluster node to which a client is connected.
In the following, it is assumed that appropriate mechanisms such as a front-end switch, or a static partitioning of clients to cluster nodes will be interposed between the cluster nodes and the network this is a logical view—in practice, if a switch is used, it will typically be duplicated in order to satisfy High Availability requirements.
Its purpose is one of balancing the load among the cluster nodes. This is discussed in more detail in section 3. It is also assumed that any of the cluster nodes will view the file system obtained through the aggregation of the logical volumes as a single entity see requirement 6, above. Only in case one of the servers crashes, other servers would take over the handling of the storage volumes that are no longer available because of the crash. Of course, this keeps the design simple and increases availability.
This scheme implies that in the case of multiple concurrent accesses to the same storage volume, they all have to be funneled through the same cluster node, which then would become the bottleneck. Thus, the scheme does not provide any enhancement in terms of performance, with respect to the situation in which one is using separate servers, each dealing with its own storage volumes.
On the other hand, this type of architecture provides a fail-over capability and implements a High Availability HA solution. However, since a key focus of the file system clustering solution is that of increasing the scalability of the solution, with respect to performance, this approach is not particularly well-suited to embodiments of the present invention. At the other extreme, it is possible to envision clustering schemes based on distributed file systems with fully shared volume ownership.
This means that any cluster member can act upon each FSO at any one time, without any static or semi-static ownership allocation policies.
Several architectures have been devised to do this. However, most of these solutions references [Thekkath ] and [Preslan ] are heavy-duty and complex and are based on complex distributed locking protocols, which are hard to develop, debug and deploy. The DiFFS proposal greatly simplifies the issues related to distributed locking and cache consistency by devising an architecture that implements the following concepts:. Only one active cluster node is responsible for the changes to a file system logical volume.
Multiple cluster nodes have read access to all FSOs and are able to serve directly read requests from clients, by accessing the SAN backend. A streamlined inter-node protocol is available to support intra-cluster coherency.
FSO references directory links can span logical volumes. FSO relocation is available and is, in fact, used to dynamically relocate FSOs from hot spots so as to achieve better scalability. A full specification of the protocols involved is available.
These are not the only concepts embedded in DiFFS. However, they are the ones that have the most relevance for the subsequent discussion, since this architecture relies on the first four and, to a lesser extent, on the last two. Examples are file reads, directory searches and requests for file attributes or ownership information. It will apply to file writes, creation and deletion of new objects, changes in attributes and so on.
This architecture envisions volumes that would be fully self-contained. In other words, each file system tree within a volume would have no physical pointers outside itself. Having logical volumes physically self-contained simplifies the model, makes integrity checking simpler and lends itself to an efficient implementation. In addition to this, it provides better fault-isolation a key attribute for a highly available platform.
The apparent negative consequences of this are the following:. Note that, despite the appearance, none of these consequences is in fact true because:. Note that since this architecture supports a single file system image see FIG. In most practical cases, it is assumed that each cluster node will manage at least one of the volume that implement the global file system.
So, the typical minimum number of volumes is equal to the number of cluster nodes. Note however that this is not a strict requirement of this architecture and that the number of cluster nodes can be greater, equal or less than the number of the volumes the form the clustered file system.
Based on DiFFS criterion 1, individual volumes are owned by a single cluster node. Such owner nodes are the only ones allowed to modify the content of the volume.
However, based on criterion 2, all nodes are allowed to access any volume in read-only mode. This implies that all read accesses to a volume can be performed in parallel, whereas any modifications to a volume have to go through its owner node. In addition to the above, this architecture divorces the logical pathname used to reference a given FSO from the actual location where the FSO physically resides.
As it will be seen, this ability allows the cluster architecture to provide high throughput. The above choices have some relevant implications. By funneling all modification requests for a given logical volume to the volume owner, it is clear that this can potentially become a bottleneck. The obvious resulting advantages are the streamlining of the protocol and the reduction in complexity. However, in evaluating how much of an impact on performance this might have, it is certainly appropriate to expect that the great majority of the traffic would in fact be made of read operations.
This has to be qualified further for the following reasons:. Caching implemented on the client side makes it possible to satisfy read requests from the client-side cache. This in fact tends to reduce the actual unbalance between reads and writes, in favor of writes. Reads are synchronous, writes mostly asynchronous. Thus, since any cluster node can directly process reads, good response time can be expected.
Read operations have the side effect that the access time for the i-node that represents the FSO being accessed has to be updated. The i-node is the equivalent of a file record in Windows NT, and is the file system data structure that stores the metadata information for the file system object it describes.
This side-effect would in fact turn any read operation into a write operation that would have to be handled by the logical volume owner. This is extremely undesirable, as it would effectively transform all read accesses into write accesses and would turn this scheme into a severe bottleneck.
However a way out relies on a less stringent criterion in updating the i-node access time and the use of lightweight, asynchronous calls to the volume owner. The access time in the copy of the i-node in the local cache could be modified, without invalidating it.
A notification would then be sent to the volume owner to do the same and to mark the i-node as dirty without requesting a flush. The master copy of the i-node would accumulate the changes and would be flushed when appropriate, thus invalidating all of its cache copies throughout the cluster. The issues outlined in the previous section provide at least an overall framework for discussion.
However, they barely scratch the surface of the problems to be analyzed in detail. This section looks at some of the specific problems to be addressed. The underlying assumptions in this architecture are the following ones:. It is assumed that an appropriate load-balancing mechanism is available to partition the clients of the cluster among the cluster nodes. This functionality might be implemented through a switch, with the partitioning done in terms of the identity of the users more on this ahead.
Load balancing is discussed further in section 4. A high-speed interconnection among the cluster nodes will be available. This will be used to support intra-cluster cache coherency and replication, as well as to relay operational requests and deliver replies to other cluster nodes. The intra-cluster interconnect is discussed further in section 4. High-availability modules will be in charge of guaranteeing the redundancy of the system and its continuous operation.
The following subsections describe the architecture in greater detail. A detailed description of the architecture requires an explanation of the structure of the file system namespace, of caching, and of locking.
As it will be clear after reading the rest of this document, the architecture defined here allows one volume to act as the root of the entire file system.
All other directories and files could be allocated within other volumes but would be referenced through pathnames starting at such a root. This is elegant, simple and adequate for the typical needs of the clients of this architecture. However, shares, exported mount point and cross-volume links can avoid this. This unique ID should not be reused, ever.
This index is valid until the volume is unmounted and is used to implement a total order relation among the volumes. This might be useful in case it is necessary to lock multiple FSOs residing in different volumes by increasing mount index values to eliminate circularities and prevent possible deadlocks.
Logical volumes are self-contained, meaning that no physical cross-references between logical volumes need exist. In other words, despite the fact that a global root volume may or may not actually lump together all the trees in the forest supported by the cluster, each tree has no pointers outside itself. However, this is not done in terms of physical storage, but rather in terms of symbolic references. Therefore, such pointers are subject to interpretation in the appropriate context and do not violate the criterion of keeping volumes self-contained this is a major departure from the Scheme implemented in DiFFS.
This attribute of having logical volumes entirely self-contained simplifies the model, makes integrity checking simpler and lends itself to an efficient implementation. They are defined in subsection 3. Because of the above, this clustering scheme is able to implement the single file system image semantics across multiple volumes. Note that logical hierarchies like the one depicted in FIG.
At the same time, it requires non-mutually-exclusive caching across cluster nodes. This is so because, even though only one node is the logical volume owner, multiple nodes may concurrently access a volume for reading.
Thus, the main resulting problem is one of making sure that data read in through the SAN backend and cached within a node with read-only access does not become stale with respect to modifications being carried out by the logical volume owner. This will be dealt with in detail in the section about caching and locking. Foreign references are requests that are addressed to logical volumes whose owner is not the recipient of the request.
Early on in the processing of these requests, the code must distinguish between the two cases. Moreover, foreign requests must also be discriminated depending on whether they are of the read or of the write type. The following is a summary of the actions to be taken in each case:. The negative characteristics of this design are the following ones:. However, by divorcing the notion of a FSO pathname from its location i. A squidgy link SL, in the following is a hybrid between a symbolic link and a hard link.
It is used to provide a symbolic reference to FSOs stored in a different volume. This directory is hidden from the NAS clients.
The values will map to a bit address space. The value 0 will be used as null. SLs can be implemented by extending directory entries so as to include a volume-ID. SLs, on the other hand, point to a different volume-ID and the i-number is replaced by a numeric value that encodes the numeric string representing the name of the referenced FSO in the RD of the target volume.
By containing a symbolic reference, SLs have no physical knowledge of details of the volume they point to. By avoiding the allocation of a separate i-node, they save storage and are more efficient. By using this approach, SLs end up having the following attributes:. The use of SLs enables the usage of one or more cluster nodes as pure metadata servers for the directory information. This can be meaningful in certain application scenarios and can be obtained by assigning no user connections to the node or nodes that should only deal with the directory hierarchy.
SLs allow divorcing pathnames from actual FSO locations. Because of this, it may happen that, as a result of node faults, either SLs end up pointing to inexistent FSOs on other volumes or that FSOs created in the RD are in fact orphans, by no longer being referenced through an SL. Note that these are not catastrophic failures, in that they cause some storage space to be temporarily wasted, but they do not compromise the integrity of the volumes themselves.
Thus they do not inhibit volumes from being remounted after a fault. At the same time, it is desirable to remove such possible inconsistencies. This can be done on the fly, as such dangling references are detected by the file system.
In addition to this, special scavenger processes can accomplish this by asynchronously removing such inconsistencies in the background, as the volumes whose integrity can be checked individually are being used. There is enough information redundancy to detect the fact that either a relocated FSO should be deleted or its should be reconnected to the owning SL.
Such scavenging will be performed in a directed fashion in case of system crashes in that it will only be applied to FSOs being modified at the time the crash occurred. In addition to this, there are some other behavioral rules that must apply to SLs, in order to make full use of them in the cluster architecture. This will be discussed in the following subsection. Also, SLs can fully generalize the notion of hard links for cross-volume references, as described in the subsequent section and in section 3.
Since there can only be a single link hard or SL pointing to a directory, the directory itself can have only one VPD. Thus, it is possible to embed a reference to the VPD within the directory i-node. The easy way to do this would be that of embedding the volume ID and the i-number in the referenced directory i-node. However, by doing this, specific information of a volume would be embedded within another one, which is something that should be avoided.
Thus, it is better to proceed according to one of the following ways:. By using a file stored in the referencing volume, that supplies the necessary mapping, as follows:. By placing the VPD backlink inside the referenced directory i-node as a pathname. This saves the need to have a separate file, simplifies updates and reduces the number of indirections.
On the other hand it needs to embed a variable-length pathname within the i-node and requires a full pathname lookup to get to the VPD. Although this is logically simpler and requires a single object to be updated to keep the reference to the VPD, it has the significant drawback that in case the VPD or any directory in its pathname is renamed, all the relocated directories that have that component as part of the stored VPD backlink, need be updated as well.
This may be an expensive operation. The first approach is a bit more complex. However, since it has no impact on renames, this is preferable. It is not expected that this will cause significant performance penalties, since the lookup of VPDs should be a relatively infrequent operation.
However, should this be the case, appropriate strategies can be devised, by implementing a hybrid scheme that stores pathnames within the directory i-nodes as in scheme 2 as a hint and relies on the first approach, should the hint be invalid. Note that this scheme only performs the caching when the relocated directory is accessed and does not embed foreign volume information within i-nodes.
Once the concept of an SL is defined, it can be usefully employed to implement the semantics of hard links across volumes. This is really beneficial, in that this is last step in hiding from the clients the notion that the global file system is implemented in terms of individual volumes.
In other words, a client will always be able to create what it perceives to be a hard link from any directory to any file in the file system, regardless of the volume they reside in. The key issue here is the implementation of the Unix link system call that NAS clients can apply to create hard links only to files. Effectively hard links can be emulated when cross-volume linkage is desired, by using SLs. It is assumed that, as in the case of deletions, the node that owns the volume where the link is to be added will always perform the operation.
In case the receiving node is not the owner, it should forward the request to the owner. The possible cases are as follows:. This is the standard behavior expected of the link system call. When a hard link to a file is to be created and the two are not co-resident, then first a hard link to the target file must be added in the RD of the volume where the target file is located, then a new SL referencing this new link should be created in the appropriate directory. Note that by simply creating a new link in the RD in the same volume, the file is not moved, and other references to the file are not affected.
When a new hard link must point to a file referenced through an SL, then the file system should create a new hard link in the RD that contains the target file the one the existing SL is pointing to.
The link count for the target file would thus be increased by 1. Finally an SL embedding a reference to the new hard link should be created in the appropriate directory. No additional functionality must exist in the file system to fully support this, with respect to the functionality that must be present to implement the basic SL functionality described in subsection 3. Handling the various cases this way results in the following useful attributes:.
Note that this completely generalizes the ability to provide a mechanism that implements cross-volume links with the full hard link semantics to NAS clients.
This way, there exist two different namespaces:. This section provides more details on operations implemented over SLs. Incoming FSO creation requests should automatically cause the creation of an SL when one or more than one of the following conditions is true:. The name used for the FSO should be chosen as described earlier. Immediately after the FSO is created, the file system software in the cluster node will create a squidgy link through the owner of the foreign volume which will then insert the new name into the parent directory.
From then on, all accesses to the FSO should occur in the local volume through the SL and write accesses would be funneled through the volume owner. A symbolic reference contained in a SL looked up to obtain a handle for a specified name within a certain parent directory. This is always a normal lookup, and the returned handle will include the object number and volume number for the required object, as specified in the directory entry.
The file system software will implement the behavior explained in section 3. The file system should also detect the inability to perform the interpretation and should distinguish the possible causes:. In case of failure, on the basis of the above, the file system should take appropriate recovery actions and should not enter a panic mode. When an existing FSO that is not physically allocated within the logically containing volume is accessed, the access should follow the SL that points to the appropriate RD.
If this is a read access or a directory access, it should proceed as discussed earlier. A write access to a directory setting of attributes, entry removal, creation of a new directory, etc. When an incoming delete request is received, then there are two FSOs that need to be modified—the parent directory and the target of the delete. Each of these FSOs may be either local or foreign giving rise to the following four cases:.
The file system software detects this case and will then i remove the link from the local parent directory and destroy the SL. This owner will then delete the link to the target itself and will reclaim the FSO space, if the hard link count to the FSO has reached 0.
An optional mechanism enabled by an appropriate configuration parameter could drive the automatic relocation of files based on the usage statistics gathered during the system operation.
In order to reduce complexity and extensive tree copies, this would probably only be applied to files and not to directories, although it could be applied to directories. This would be done through a mechanism similar to the one outlined for the creation of an FSO in a foreign directory. A background task would perform this, so as to allow the further distribution of hot spots and the balancing of the load across the cluster nodes.
The steps involved in relocating each file would be the following ones:. An important caveat is the following: because of the way SLs work, a file with multiple hard links cannot be transformed into a SL pointing to a different volume because otherwise it would be necessary to trace all the links, to transform them into SLs to the same file and this is potentially a full-scale search within the volume. Thus, files with more than one hard link would generally not be relocated at all, although this rule could be relaxed if the existing links are readily detected, such as when they are all within the same directory.
When a cluster node tries to gain write access to a file which has already relocated and is represented by a SL in the original containing directory, no further attempt is made to relocate the file to a volume owned by the local cluster node. The reason is to prevent continuous relocations and that the first relocation already served the purpose of avoiding the concentration of multiple heavily accessed files within the same volume.
Besides, assuming the load balancer already bases connections on user identities, the file should have reached its most appropriate destination. Note that the automatic relocation scheme is based on knowledge of the nodes through which the users connect to the cluster.
In order for the relocation to add value beyond the pure scattering of files across multiple volumes which is itself valuable , it is desirable that the file relocation be stable. This means that a given user should always connect through the cluster through the same cluster node. A further cluster configuration option which might be disabled entirely or conditionally allowed on the basis of certain thresholds, based on configuration settings , subject to certain thresholds in file size could allow automatic file relocation on the first write access.
If the write access entails modifying the content of an existing file, then two cases may occur:. The mechanisms outlined here effectively perform selective relocation of files within the cluster. This has the merit of spreading out files to the various volumes in an automatic fashion, without giving that perception to the users. In order to guarantee adequate performance, along with data consistency and integrity in the cluster, appropriate caching in each node must be supported. This should make sure that the cached data in a node with read-only access does not become stale with respect to modifications being carried out by the logical volume owner.
This issue is closely tied to that of internal system locking and calls for the ability to use appropriate communications mechanisms to request the update of a volume with new data and to make sure that cached data and metadata in the cluster members are promptly discarded, when stale. This means that it is necessary to:.
The system-level lock manager can be integrated with the cache management since the breaking of a read-lock because of a write and the flushing of the relevant cache page are in fact related events. The actual lock manager for a volume is run within the node that is the owner of that volume.
The architecture expects two caches to be logically available. Another one is the sector cache, operating on the SIM board. Local requests would be handled essentially in the metadata and sector caches. However, foreign requests should be handled differently. Foreign write requests would totally by-pass the local caches see FIG. Foreign read requests see FIG. The scheme to be used for the sector cache should allow foreign volume sectors to be cached in read-only mode when a read-lock is valid on them.
The requestor should block until it receives both the sector and the read-only lock for it. There are two cases:. If the volume owner does not have the sector in its cache, it should grant the lock, should record the cluster node that requested it and the sector number and should let the SAN backend satisfy the request. At this point, the requestor would get the lock and should wait for the sector to be available from the SAN.
Then it would satisfy its client's read request. If the volume owner has the sector in its cache, then there are two subcases:. If the sector is not write-locked, the volume owner should reply by granting the read-only lock and sending the sector content with it. The requestor then would satisfy its client's request and would drop the sector, whenever it is returned by the SAN backend or should abort the SAN command.
If the sector is write-locked, then the requestor should be stalled until the lock is granted, which goes back to case a. When the volume owner needs to write-lock a sector, it first should break the outstanding read-locks in the other nodes. This has also the effect of invalidating the cache entries for that sector in the other nodes. At this point, any node needing that particular sector should go back to execute the above algorithm.
However, every other node in the cluster can potentially cache read data from volumes which it doesn't own, and so it is necessary to have a scheme to ensure that when the volume is written to by the owner all the cached data on other nodes can be invalidated in order to maintain data consistency.
In a typical embodiment of the present invention, every node in the cluster contains a sector cache lock manager. This manages the caching of data in all the sector caches in the cluster from the volumes which are owned by the node. The sector cache is made of cache pages 32 Kbytes in size, in a typical embodiment and it is these pages which are managed by the lock managers. Each cache page managed by the lock manager can be in one of two states:.
Read-locked—this page is cached in the sector cache of at least one of the non owner nodes. Write-locked—this page is cached in the sector cache of the owner node and may contain dirty data the data in the sector cache is potentially newer than the data on the disks. There is a question as to how many cache pages an individual lock manager should be able to manage. Obviously in the worst case all nodes in the cluster could have their sector caches full of different blocks of read data from a single volume.
One solution to this is to specify the memory on the lock managers wants to cope with this worst case. There are three different sector cache operations that can occur. The implications on the lock manager for each of these are discussed below. When the volume owner receives a request to read data on a volume it owns then it is free to read the data from the volume into its owner sector cache without taking out any locks or checking the state of the lock.
If the page was previously write-locked, then it will remain write-locked. When a non-owner receives a read request it first checks to see if it already has a read-lock for the page. If it has, then it can just read the required data. If not, then it must apply to the lock manager for a read-lock.
There are two possible outcomes of this request:. If the page isn't currently cached anywhere in the cluster, or is read-locked by another node or nodes, then the lock manager will mark the page as being read-locked by this node and grant a read-lock to the node so that it can read the data from the volume into its sector cache.
Privacy policy. Run an antivirus scan. Open an elevated Command Prompt window. To do this, select Start , enter cmd , and then press Enter. In the search results, right-click Command Prompt , and then select Run as administrator. Restart your device, and try the action in Word again. If the error still occurs, try the next method. Scroll down to Related settings , and then select Advanced system settings. After many hours of fighting it, I have a solution, but it doesn't feel like the right one.
Would still like an explanation if anyone understands this better than I. Sign up or log in Sign up using Google. Sign up using Facebook. Sign up using Email and Password. Post as a guest Name. Email Required, but never shown. The Overflow Blog. Podcast Making Agile work for data science. Stack Gives Back Featured on Meta. New post summary designs on greatest hits now, everywhere else eventually.
Related Hot Network Questions.
0コメント