Monday, May 4, 2009
Folders in doc libraries are metadata cries for help
On twitter, @PirateEric is lamenting the fact that users INSIST on creating folders.
@PirateEric -”another folder ordering conversation, i DETEST folders in libraries aaaaaa!”
@LoriGowin – “let me know how that goes for you. It’s like pulling teeth to get people to think metadata, not folders!”
@EUSP – “It’s all in the presentation. Show how to solve the problem and THEN talk about it. Show, tell… Show… tell.”
@SaraHaase – “Folders in doc libraries are metadata cries for help.”
I agree. Even if people don’t know about metadata, they are asking for it when they create hierarchical folders. That’s REALLY what they are looking for… structuring their information so that it can easily be found.
Until there’s a decent way to have virtual folders that appear to end users through the Windows Explorer interface, users will continue to create hierarchical folders.
I don’t know about that, Dan. I think that teaching them groups still gives them the satisfaction of clicking on a plus sign. It’s a matter of teaching the entire solution and not just wagging your finger and saying, “don’t use folders or the booggie man will get you.”
The issue that I see presenting itself is SP being deployed without proper governance / guidance in place. One of the great things about SP is its viral nature. Unfortunately in the case of folders and explorer views, it can also be its demise as information gets shuffled into yet another digital dumping ground of folders embedded within folders.
With proper guidelines put in place, and users understanding what content types are as well as metadata and views, it becomes an information architects dream to watch user adoption go full steam ahead.
Bottom line, this is a BIG change for the end user and change is not easy. We all accept and deal with change at different levels and in different ways. I would suspect many of the contributors accept and even embrace change. So it is hard for us to not completely understand why everyone else just doesn’t get it or is soooo slow to change.
I have been dealing with going folder-less (use metadata) for some time, hoping that someone’s presentation or view would be the panacea. No luck.
I have come to believe the underlying issue is change itself and dealing with people’s ability and process for changing.
To make matters worse, for those that use SharePoint we live in two different worlds. We live in the present and the future.
The future world is SharePoint. A place where you are no longer restricted by physical location, but can organize in new ways never done before, which can be exciting to some and scary at first glance to others.
The present world is Windows operating system, where folders are very alive and well. We have grown up organizing our digital world into folders, sub-folders, sub-sub-folders etc. And that still exists.
So we are asking the end user to change, but not completely change. When in SharePoint go folder-less (Metadata), when on your computer folders. Conflict city!!!
People want consistency and the change we are asking of them is conflicting, a challenge, new and unknown. Many of these end users have just begun the journey and do not have the understanding or experience that you have.
This process will take time, lots of education, and lots of patience.
We have a small cult following of using folders because of the way it carries it into the ‘connect to outlook’
Content types, as important as they are, simply are not convenient out of the box. It does require training and does require setup at the site level. So from an end user standpoint they have no clue. They only see that they can create a new folder. I would like to see Microsoft move content types, out of the box, up and more into the administrators and end user’s setup experience. Maybe redefine folders as labels, like Google does, and give the label some content type ability, to create on the fly. In the mean time training on the setup will remain critical.
Information Worker End Users should have nothing to do with content types. They shouldn’t even know they exist.
In order to get user adoption, the words metadata, content type, information architecture, SharePoint… should never be used. They don’t care that it is SharePoint or the doghouse.
Show a solution that solves a business problem. The solution should be the smallest chunk of process possible. When that works, you can offer a little more.
Don’t put a bushel of corn on the table when all they asked for was a simple side dish.
Mark, you’re right that the solution is the end goal and that the technical details would only defray from that goal. There are several ways to make that happen where the IW utilizes the system if setup properly to tag things without knowing it.
I’d say as long as things are put in place properly and then IWs are shown a process, you’ve solved the majority of the problem and may even in turn eliminate folder user :)
What about large files and large number of files? Sharepoint has some issues when it comes to dealing with more than 2000 files in a library if you do not spit them up into multiple folders and so on.
I love the idea about having a folderless structure, but my experience is that the only thing that seems to work for both users and SharePoint is to utilize a combination of the old ways (folders) and the future way (metadata) and virtual folders.
I think that ricknology has a good point, things have to change for this to be accepted among the users, Google is leading the way so far.
I don’t agree that “number of files” is a reason to create folders. Creating a folder is not going to help if the return results is more than 2000. You’re still going to have to create views to stay within the limit. — Mark
Hi Mark,
That is true, however I often find that creating views that can display quite large numbers of documents is mostly about doing the opposite of what I would like to do to make it possible to overview; that is grouping multiple levels and/or showing many documents at once.
I would like to have a flat file structure that was fast and only relied on metadata. This is possible to some extent by using indexing on flat folder view for large collections of documents.
But even how I look at it, Sharepoint seems to have its limits that give you quite a bit slower performance.
I would love to hear about ways to work around that,
Thanks for having a great site!
Hi Marc, I have to agree with Tilgmann that sometimes the only workable solution is to mix Folders with metadata.
The problem isn’t only user acceptance of metadata, in one case I had to create multiple folders because the client wanted very specific permissions implemented for each and every kind of documents. While the end-users loved the metadata/group/filter features, the hierarchy wouldn’t budge from their permissions requirements. As metadata don’t support permissions, folders were mandatory.
Some might say it’s a governance problem, I would agree to some extent but the fact remains that some services/businesses need more secrecy than the average service.
-Jonathan
One of the challenges is that Microsoft recommends (in their capacity planning documents) the use of folders in libraries that contain more than 2000 items for performance reasons. The alternative is to split documents across multiple libraries but that renders the popular content query web part all but useless and can influence search relevance.
Sharon,
In your last statement, this can be addressed with content types. By building content types and applying them to document libraries, your are able to use the content query to view only the documents you would like to show based on content type. Also, for searching you can create searches based on content types and thus find documents across libraries with the same content type. This can be a very nice way to address viewing and finding documents across multiple libraries.
Hmmm I’m really unsure what to do now. I am about to populate a Documentation Center and decided to use multiple Libraries for each product and know there will be a large amount of files.
In each Doc library I was going to create folders for each version of the Software release. I am using metadata in each library as well.
Now I am concerned about search, we are using the GSA. I am also just starting to familiarize myself with the CQWP so I am concerned if I ever need to use that, I will not be able to becuase I have not set up the center properly.
Looks like I need to do a little more research on the CQWP. As well as reconsider the folder thing. Any further comments advice etc.. glady accepted :)
At least within the library itself, the only reason I’ve used folders is for specific permission issues. Otherwise, I’ve found that using grouping within views can provide the same functionality that folders do. At least when viewing the library.
My mantra for many years has been “no folders”.
We’ve disabled the Add Folder option on many of our sites to encourage the use of “columns” and “values”.
@Tilgmann I’m wondering when is there a case when someone wants to see 2000 files at a time.
In my experience my site owners find that the grouping option along with a limit of the number of files shown on a page is a good solution.
We find that people are typically interested in the most current content, but giving them the ability to get to the other documents easily is the key.
Mark, I completely agree with you about putting it in the terms that people understand. Use words like “categories” or “tags” and show them how to filter their views in multiple ways – folders don’t give you that option (unless you use the “show items without folders option” but I don’t want to derail more).
People are getting used to the concept of tagging with tools like Flicker, Delicious, Twitter and even Windows Vista.
Filters and views are very powerful, and the great thing about them is that they generate a URL so you can create a link to any view and post it in your quick launch or your top tabs. That’s been very useful for my site owners.
Deborah,
I would look at using metadata and content types to address what you are doing.
Personally, I look at using separate libraries from a access stand point. So if each product is going to have very different access rights then a library for each might be the answer. If not then one library could do.
One way to manage your library is through metadata and content types.
Content types is a great way to create consistency around document purposes. For example if you have a requirements document then that could be a content type. Now you can have specific metadata, policies, work flows, and a document template all associated. This would then allow you to create searches specific to that content type. Also, those that create requirements would be using the most current template since it will be a selection under new on the document library.
There is so much that can be done with content types and metadata. But the structure needs to be figured out first.
Peter,
These documents are not created inside Sharepoint they are written by the tech writers outside of Sharepoint and then uploaded. So ther is no template. I suppose that is another hurdle. To get them to work directly in SharePoint with a specified. That I would have to sell hard. Until I can get there…
Permissions are not going to vary much at all here. I am mostly concerned about the amount of documets we will be putting in one library. I will easily exceed 2000
If I create multiple views will that help? As in does each view allow you to add more than 2000 items into the library? As long as ther is no performance degradation than I can just limit the number of files shown?
we do use filters and views and metadata quite a bit throughout our sites. I Agree with you about the use of Content Types. I need to look at that at the site level.
I am moving over our docs now and adding metadata to each one, painful and slow. As we go forward I am forcing the users to add metadata to each new file. Thaks for your input greatly appreciate the feedback!
Deborah – content types are used for uploading as well as creating new docs in the library. The only missing part is getting outside users to use the correct doc template which has to be managed separately. But I would assume external writers would be happy to accept the template you provide them and all doc properties will upload as normal.
Exceeding 2,000 items in a library isn’t a problem. The claim is that a document library can store up to 2 million docs. I haven’t had a client go anywhere near that, but I do the same as Peter and encourage them to use libraries to control permissions rather than attempt per-item permissions within a single library. That alone seems to guarantee reasonable numbers (comparable to folders on a file share.)
Default views limit items returned to 100 per page to control performance. You can go higher than that – I’ve seen variable results. Clients with good bandwidth have gone up to 700 items per view. Clients with bad bandwidth suffer beyond 150 items per view.
A lot of confusion arises from Microsoft’s white paper which recommends using folders to manage more than 2,000 docs per library. I’m with everyone here and discourage the use of folders. But… the challenge is if you find performance degrades after 2,000 docs and you complain to Microsoft Support, they may refer you to that white paper and suggest you implement folders. You either follow their recommendation or split the documents across multiple document libraries.
GSA has a connector for SharePoint and can index docs and their properties… tho’ I would always suggest using SharePoint’s own search first as GSA has its own set of challenges.
Most common use of CQWP is to display contents of a list or library on a different site within the site collection. You can filter by any property (column) and do all sorts of other funky stuff. If you have multiple libraries then, out of the box, you’d need multiple CQWP to display all the content. But that’s not the end of the world, you get web part titles instead of grouping titles within a single web part. It takes up a bit more screen space but I find users prefer it for quick access to specific information.
Finally, a quick tip if you are in the process of importing lots of docs. Dump them all in first. Then switch to datasheet view to apply metadata. It behaves like Excel (with good reason) and is a far quicker way of applying metadata to existing docs.
If you want to chat through this further, happy to have a conf call to discuss. Contact details can be found on my web site (click my name here).
> Dump them all in first. Then switch to datasheet view to apply metadata.
Another way to do this is to setup default metadata settings for a group of documents.
Pull the first set of documents in and the default metadata will be attached. Reset the default settings for the second batch and pull them in. Rinse and repeat.
Quick, cheap way to attach metadata to large batches of documents.
Mark
Ooo excellent suggestion by Mark and by far the better method when bringing in a large quantity of docs. Takes a bit more organisation up front but well worth it and less effort than manually applying metadata afterwards
One final tip. When importing existing docs, keep an eye on the document titles. People often create documents by starting with an existing one. When you create a document from a blank template, unless you specify otherwise Word will insert the first line as the title. But it’s a one-time offer. If someone creates a document by basing it on an existing one, the title will remain the same unless the user explicitly goes in and changes it. And that means the title can be completely inaccurate. It’s a real gotcha for skewing search relevance.
Good luck with the migration! :-)
Sharon, Mark, Peter,
Thanks I was able to use info from all 3 of you. I set up the Doc Center with ONE library with multiple views and metadata (Product name, Product version and Document type)and groups.
When I upload a single document, I get prompted to select the necessary metadata (product name, Product version and Document type). If I upload multiple docs then I have to go into datasheet view to apply the metadata.
So unless there is a better way, than I think I’m good. I have already transferred docs over and am happy with the way it is displaying the data. Any additional tips are always welcomed.
Hi all,
A question, does none of you experience any performance issues when storing two or even four thousand files in a flat file structure?
I have also heard that you can store as many as 10 000 documents in a flat structured library. But on the other hand I have first hand experience on how performance can start to decrease a little bit after a 1000 documents and that it starts to get really noticeably when you exceed 2000.
I seam to be the only one here that still think that folders still is a necessary evil that can help to increase performance.
I only use folders if:
1. I set up a new library and now that the number of documents will be high;
2. I have a library with a high number of documents and slow performance; or
3. I have to set unique permissions to a large collection of documents.
I think that Libraries are good if you have a different department whit a unique set of content classes and permissions.
My experience is that it should not matter if you have folders or not, as long as you have setup a logic metadata structure and views. The views even have the options checked not to show folders. If you have spread out your documents over multiple libraries on the other hand—then you will have to use something like the CQWP to display the files.
Either way, here is a tip that is good to use if you ever need to send a link to collection of files to someone:
Use the filtering in the top of the columns than copy the URL.
The URL gets a bit messy but it is quick way to isolate a custom group.
Regards,
Tilgmann
Hi Tilgmann
MS doesn’t recommend the use of them without reason so yes, they may be the only solution if you need to improve performance of a large doc lib. But I’d still avoid them if possible rather than introduce them by default, if that makes sense. And in the current version, that usually means multiple doc libs rather than one big one.
For URLs, I create custom views to keep the URL relatively clean. Plus one tip for naming libraries and views (the title) – always start with one word, then go back in and modify. The first name you use will be the URL, spaces (%20s) and all. e.g. if doc lib is for ‘Press Releases’, start with ‘pr’ and that becomes the URL, then rename the title afterwards to be ‘Press Releases’. Same for creating views. :-)
And whilst we are in Tip Alley :-) If performance is starting to slow, one view I create is ‘Select a view’. I add a file/item to the library/list with the title ‘Select a view’ and description ‘From the menu in the top-right corner of this library’. Then create a view, filtering by “‘Title’ contains ‘Select a view’” and make it the default (there’s a checkbox just below the title and URL for the view). It’s great for people who hit the library and just want to create or upload a doc versus find one. Those who want to look for something then select the view they want for instant filtering.
Cheers and regards
Sharon.
Hi Sharon,
Thanks for your tips on URLs and naming conventions, it is always a fun puzzle to get names that are short and meaningful. And your solution for ‘Select a view’ is great, Thanks for sharing!
Hi Mark,
in your post of may 5th you mention that for uploading mutilpe files the best is to upload all then edit metadat in the ‘datasheet view’. I am able to switch to this view, but it is read-only. I read in some help that it requires Acces, so I installed this, but still can’t edit. Any ideas what I need to do/enable to use this way? Thanks a lot!
Anja