Google App Engine (GEA) finally got Java support. That's good. This is leaving me with no more excuses for delaying my Shogi Tools work.
So I got a grip and wrote some more code. Right now I am focusing on turn-based Shogi server.
I wanted there to provide the functionality to upload a user's picture.
But how to achieve this in GAE? GAE doesn't allow you to write files on the discs. How to do this fast and easy? I remember using in some of my earlier projects a ready to use Java library that made file uploading very easy for me.
Apache Commons FileUpload
Thank God for people from Apache.org. I went to their site http://commons.apache.org/fileupload/. The folks prepared a library to add robust, high-performance, file upload capability to our servlets and web applications. I decided to see how would it play with GAE.
Below there is a description of the process of creating a Java web application, under Eclipse environment, which uses commons FileUpload.
Below there is a description of the process of creating a Java web application, under Eclipse environment, which uses commons FileUpload.
Example Google App Project
I assume you have downloaded and installed Google Plugin for Eclipse. In my case I used Eclipse 3.4 (and the dedicated plugin).
We'll start off with creating new Google App Engine project.
Open up your Eclipse. Then create new GAE project.
You can do this by clicking g+ icon on Eclipse toolbar (
) or choosing File->New->Other->Google->Web Application Project. "New Web Application Project" dialog shows up.
) or choosing File->New->Other->Google->Web Application Project. "New Web Application Project" dialog shows up.
For the purpose of this "tutorial", provide the project name ("UploadTest"), package name ("org.fbc.uploadtest") and uncheck the GWT support. After clicking "Finish" GEA Eclipse
plugin creates a new project for us with a structure similar to the following screenshot:
HTML page for file upload
Next, we create a HTML file which enables the user to pick up the files to upload to the server.
Do this by replacing contents of (generated by GEA plugin) index.html. Open it for editing and paste the following content:
Do this by replacing contents of (generated by GEA plugin) index.html. Open it for editing and paste the following content:
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>GAE file upload test</title>
</head>
<body>
<form name="filesForm" action="uploadtest" method="post" enctype="multipart/form-data">
File 1:<input type="file" name="file1"><br>
File 2:<input type="file" name="file2"><br>
File 3:<input type="file" name="file3"><br>
<input type="submit" name="Submit" value="Upload Files">
</form>
</body>
</html>
There are few key points here:
- form's attribute "method" is set to "post"
- form's attribute "enctype" is set to "multipart/form-data"
- form's "action" points to the servlet that will handle the upload (yet to be created)
- input filed type is "file"
Configure Apache commons in Eclipse
First, if we haven't done it yet, we should download FileUpload and IO libraries from Apache's page. I decided to use the latest (1.2.1) version of FileUpload (http://archive.apache.org/dist/commons/fileupload/). The version is dependent (see http://commons.apache.org/fileupload/dependencies.html) on version 1.3.2 of IO library, so I downloaded exactly this version (http://archive.apache.org/dist/commons/io/binaries/commons-io-1.3.2-bin.tar.gz).
After unpacking the contents look for commons-fileupload-1.2.1.jar and commons-io-1.3.2.jar. Copy the files to war/WEB-INF/lib directory (see the project structure image above). This directory contains all the java libraries that will be deployed to appspot server for your app to use them.
Now, for your project to "see" them, you have to add the to the project's build path. I do this by right-clicking on the projects node in the project explorer, choosing Build Path->Configure Build Path (see the image below).
After unpacking the contents look for commons-fileupload-1.2.1.jar and commons-io-1.3.2.jar. Copy the files to war/WEB-INF/lib directory (see the project structure image above). This directory contains all the java libraries that will be deployed to appspot server for your app to use them.
Now, for your project to "see" them, you have to add the to the project's build path. I do this by right-clicking on the projects node in the project explorer, choosing Build Path->Configure Build Path (see the image below).
Then I go to "Library tab" (1), click Add JARs (2), indicate desired files in "war" directory (3) and add them to the build path (4). Look below.
The libraries are now in the project's classpath. We can start creating the servlet.
Upload servlet
We are ready to send the file. Now we must prepare ourselves to receive it on the server side. For that purpos we change the contents of "UploadTestServlet.java". GAE plugin generated this class for us in src folder and org.fbc.uploadtest package (see the picture with project structure).Our upload form (index.html) says it sends the data to the server with "post" method. So, in our servlet we must provide doPost method. Here is the code:
public void doPost(HttpServletRequest req, HttpServletResponse res)
throws ServletException, IOException {
try {
ServletFileUpload upload = new ServletFileUpload();
upload.setSizeMax(50000);
res.setContentType("text/plain");
PrintWriter out = res.getWriter();
try {
FileItemIterator iterator = upload.getItemIterator(req);
while (iterator.hasNext()) {
FileItemStream item = iterator.next();
InputStream in = item.openStream();
if (item.isFormField()) {
out.println("Got a form field: " + item.getFieldName());
} else {
String fieldName = item.getFieldName();
String fileName = item.getName();
String contentType = item.getContentType();
out.println("--------------");
out.println("fileName = " + fileName);
out.println("field name = " + fieldName);
out.println("contentType = " + contentType);
String fileContents = null;
try {
fileContents = IOUtils.toString(in);
out.println("lenght: " + fileContents.length());
out.println(fileContents);
} finally {
IOUtils.closeQuietly(in);
}
}
}
} catch (SizeLimitExceededException e) {
out.println("You exceeded the maximu size ("
+ e.getPermittedSize() + ") of the file ("
+ e.getActualSize() + ")");
}
} catch (Exception ex) {
throw new ServletException(ex);
}
}
The code should be quite self explanatory, so I won't dwell on this (although I am open to questions :-) ).
I'll just point out few things. I marked two places in the source bold.
The first line:
upload.setSizeMax(50000);
sets the file size limit. When the user tries to upload a file larger then (in our case) 50000 bytes, an exception occurs. When setting the maximal size remember about the GAE limits and quotas.
(If you paste doPost to the class, Eclipse should automatically add imports the code uses. If not so, do add the following lines manually:
import org.apache.commons.fileupload.FileItemIterator;
import org.apache.commons.fileupload.FileItemStream;
import org.apache.commons.fileupload.FileUploadBase.SizeLimitExceededException;
import org.apache.commons.fileupload.servlet.ServletFileUpload;
import org.apache.commons.io.IOUtils;
)
The second line
fileContents = IOUtils.toString(in);
Now you have a String containing uploaded file's data waiting for you to do whatever you want to do with it.
In my example I just output the string to the browser (so when you test it it's best to use text files).
Testing the code
All is set and ready. Just run the code. I do this by right-clicking on the project root node in the project browser and choosing Run As->Web Application.
All is set and ready. Just run the code. I do this by right-clicking on the project root node in the project browser and choosing Run As->Web Application.
On the Eclipse console, if everything went fine, you should see the following message:
The server is running at http://localhost:8080/
So, open up your favourite browser and go to the address. Choose some files (remember the size limit) and submit the form.
In my case (I chose log4j.properties from src folder), I got the following message in my browser:
So, everything works fine. The browser uploaded a file to our application, which now could store the file in the GAE datastore.
Hope this info is helpful to anyone.
See you next time,
fat bold cyclop
P.S. I left working example on http://100.latest.kbguesttbook.appspot.com/.
So, open up your favourite browser and go to the address. Choose some files (remember the size limit) and submit the form.
In my case (I chose log4j.properties from src folder), I got the following message in my browser:
--------------
fileName = log4j.properties
field name = file2
contentType = application/octet-stream
lenght: 1063
# A default log4j configuration for log4j users.
#
# To use this configuration, deploy it into your application's WEB-INF/classes
# directory. You are also encouraged to edit it as you like.
# Configure the console as our one appender
log4j.appender.A1=org.apache.log4j.ConsoleAppender
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.layout.ConversionPattern=%d{HH:mm:ss,SSS} %-5p [%c] - %m%n
# tighten logging on the DataNucleus Categories
log4j.category.DataNucleus.JDO=WARN, A1
log4j.category.DataNucleus.Persistence=WARN, A1
log4j.category.DataNucleus.Cache=WARN, A1
log4j.category.DataNucleus.MetaData=WARN, A1
log4j.category.DataNucleus.General=WARN, A1
log4j.category.DataNucleus.Utility=WARN, A1
log4j.category.DataNucleus.Transaction=WARN, A1
log4j.category.DataNucleus.Datastore=WARN, A1
log4j.category.DataNucleus.ClassLoading=WARN, A1
log4j.category.DataNucleus.Plugin=WARN, A1
log4j.category.DataNucleus.ValueGeneration=WARN, A1
log4j.category.DataNucleus.Enhancer=WARN, A1
log4j.category.DataNucleus.SchemaTool=WARN, A1
So, everything works fine. The browser uploaded a file to our application, which now could store the file in the GAE datastore.
Hope this info is helpful to anyone.
See you next time,
fat bold cyclop
P.S. I left working example on http://100.latest.kbguesttbook.appspot.com/.
Thanks a lot !
ReplyDeleteDo you know where I can find resources for File Upload with GWT + GAE ?
Sorry, I have no knowledge of GWT yet.
ReplyDeleteIt looks like you do not send any parameters with your uploaded file. I have some parameters being uploaded with similar code to yours, however I have:
ReplyDeleteFileItemFactory factory = new DiskFileItemFactory();
ServletFileUpload upload = new ServletFileUpload();
Then I can call upload.parseRequest(request) to get a List of FileItem objects, however in GAE this works unless you actually upload a file, then it will throw a file system access exception.
So I can take out the factory and just do this:
ServletFileUpload upload = new ServletFileUpload();
But in this case, I get a 'No FileItem Factory has been set.' exception...
Any thoughts on how to pass both a File and some parameters in the same doPost ?
If I get You right You post some form fields and do a file upload at the same time.
ReplyDeleteIf so, the following should use FileItemIterator:
ServletFileUpload upload = new ServletFileUpload();
res.setContentType("text/plain");
FileItemIterator iterator = upload.getItemIterator(req);
while (iterator.hasNext()) {
FileItemStream item = iterator.next();
InputStream stream = item.openStream();
if (item.isFormField()) {
// I have a field
} else {
// I have a file
}
It works for me. Hope this helps.
First of all, Thanks for the tutorial, but I want to know something: I need to make an app like this one but I don't want to host the files in Google App Engine, I just want the "upload form" app in GAE but I need to host the files in another location, is this possible? What do I have to change? Thanks, my mail is: hyllier@gmail.com please contact me...
ReplyDeleteHi,
ReplyDeleteI wrote an example to show how to upload files to GAE/J using GWT. Source code is available.
Check here:
http://puntosoft2k.appspot.com/gwt_gae_example.html
It's very useful.Thank for author.
ReplyDeletegood post. I followed it and was able to upload a file to the datastore. Now I want to be able to download the file. Do you have code for that ?
ReplyDeleteI tried to use the fileoutputstream but was getting exception that it is not supported.
Here is an example servlet that retrieves image from the database:
ReplyDeletepackage org.fbc.shogiserver.actions;
import java.io.IOException;
import java.util.logging.Logger;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.fbc.shogiserver.dao.UserDAO;
import org.fbc.shogiserver.entities.Picture;
public class GetImage extends HttpServlet {
private static final Logger log = Logger.getLogger(UpdateUserInfo.class
.getName());
/*
* (non-Javadoc)
*
* @see
* javax.servlet.http.HttpServlet#doGet(javax.servlet.http.HttpServletRequest
* , javax.servlet.http.HttpServletResponse)
*/
@Override
protected void doGet(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException {
String nick = req.getParameter("nick");
log.info("nick = " + nick);
Picture pict = UserDAO.get().getUserPicture(nick);
if (pict != null) {
log.info("pict is found, putting it to the writer. ("
+ pict.getContentType() + ", " + pict.getFileName() + ", size=" + pict.getContent().getBytes().length);
resp.getOutputStream().write(pict.getContent().getBytes());
resp.getOutputStream().flush();
} else {
log.info("no file found");
resp.sendRedirect("images/noImage.jpg");
}
}
}
You can use it to show the image on JSP page by giving the following src value:
src="/getUserImage?nick=fatboldcyclop"
UserDAO.get().getUserPicture(nick) reads the stored file from the database under given user nickname.
Hope it helps.
please help,
ReplyDeleteI want to store an Image in google app engine's datastore using Java, but I don't know how to.
I have uploaded it using first example in this blog, but then I want to store and retrive it from the datastore.
my mail id is: nitinjoshi85@gmail.com
Thanks,
Nitin Joshi.
If you managed to upload the picture you probably did it with something simillar the following code.
ReplyDeleteAs I understand it, you need to upload a file to GAE, store it and then show it on HTML page.
So you need the following chunks of code:
1. Reading the file from request.
2. Store the contents in the db.
3. Retrieve the contents and make an image out of it.
I don't know how far did you manage to go with this so I review all the steps.
Ad 1)
In my case I use Apache Commons to read the file from the request and convert it to byte array/string.
//...
FileItemIterator iterator = upload.getItemIterator(req);
FileItemStream item = iterator.next();
InputStream in = item.openStream();
//...
byte[] contentBytes = null;
try {
contentBytes = IOUtils.toByteArray(in);
log.info("contentByte size " + contentBytes.length);
} finally {
IOUtils.closeQuietly(in);
}
contentBytes now holds the file.
(the whole servlet at http://shogitools.googlecode.com/svn/ShogiServer/src/org/fbc/shogiserver/actions/UpdateUserInfo.java)
Ad 2)
In my app I have a persistent class to store user images. It holds the file itself
as well as "descriptive" data (file name, size, etc.)
@PersistenceCapable(identityType = IdentityType.APPLICATION)
public class Picture {
//...
@Persistent
private Blob content;
//...
}
(The rest of the class here: http://shogitools.googlecode.com/svn/ShogiServer/src/org/fbc/shogiserver/entities/Picture.java)
PersistenceManager pm = PMF.get().getPersistenceManager();
result = new Picture(screenName, fileName, contentType, content);
try {
pm.makePersistent(result);
} finally {
pm.close();
}
in the constructor the content field is set to
new Blob(content.getBytes("UTF-8")) // if we have the file stored a String object
or
new Blob(contentBytes) // if we have the file stored in byte array
(more in http://shogitools.googlecode.com/svn/ShogiServer/src/org/fbc/shogiserver/dao/UserDAO.java)
Ad 3)
To show the image on the web page I use ordinary IMG tag, but in the SRC attribute I give a servlet serving the image:
< img src="/getUserImage?nick=myNick" >
The servlet getUserImage get method reads the file (given by nick argument) and sends it to the response:
String nick = req.getParameter("nick");
Picture pict = UserDAO.get().getUserPicture(nick);
if (pict != null) {
resp.getOutputStream().write(pict.getContent().getBytes());
resp.getOutputStream().flush();
}
Let me know if it helps.
how can i download an uploaded file? I've seen many examples on how to upload, but none on how to download content.
ReplyDeleteThanks, nice tut btw
thanx a lot.
ReplyDeletenice work.
@anonymous
ReplyDelete> how can i download an uploaded file?
Please, see ad 3) in my latest comment:
3. Retrieve the contents and make an image out of it.
I showed how to "feed" the source of an image with a servlet that reads the data from the database.
You can use exactly the same technique to make a href link.
I really like your detailed instructions here. However, when I tried to follow them I got the following error message for the UploadTestServlet class:
ReplyDeleteorg.omg.CORBA.portable.InputStream is not supported by Google App Engine's Java runtime environment
Maybe I will be better off trying to upload a file using the google web toolkit.
I remember having similar problem when at first I used DiskFileItemFactory. It is not allowed on GAE (writes to filesystem).
ReplyDeleteThe code from the post is compiled and deployed on http://100.latest.kbguesttbook.appspot.com/. It works fine. I'll try to think what may cause your problem.
My first thought: you import wrong InputStream.
You should import java.io.InputStream.
As I recall some IDE (Eclipse for example) make imports for you when you paste some code. That might be the case.
Regards,
fbc
wwe
ReplyDeleteMany many Thanks
ReplyDeletehello my friend can i using common file upload with struts 1.x and file form ??
ReplyDelete@lukasz
ReplyDeleteThere are problems combining Struts 1 and Apache Commons FileUpload reported (http://commons.apache.org/fileupload/faq.html#howto-parse-in-action).
On the other hand, I remember (I haven't been doing Struts 1.x project in years now) Struts handled file upload itself. Try http://www.vaannila.com/struts/struts-example/struts-file-upload-example-1.html (I didn't check it myself, though...).
Powodzenia, przyjacielu ;)
Interesting blog, i usally be aware all about all different kind of sofware. i am online all the time, and this action allow me to see a site costa rica homes for sale and i like it too much. beyond all doubt without my computer i never would have seen this site too.
ReplyDeletegood presentation. do you work on GWT/AppEngine? We are a startup based in Austin,TX and are looking for some urgent help on our development activity with our product. If you are interested in any parttime/fulltime feel free to email me back.
ReplyDeleteYour one looks like temporarily holds data.Is it possible to store a file in the database to a specific path in my project (like in the WEB-INF directory) using your method because i need other files to use the uploaded file.
ReplyDelete@anonymous
ReplyDeleteI am not sure I understand what you want to achieve.
In presented solution uploaded files are stored in database for further use. You can find them by name or anything else you also put to the database.
Please, explain "i need other files to use the uploaded file." - maybe I could be more helpful then ;-)
Sorry for being unclear.Ok after i upload the file i want it to be saved to the war folder i do not know if thats possible.I want this way because i want another html page to read this file and generate a new page.
ReplyDeleteNice explanation and a great code to fetch the image via the code. Fetching the image is taking a bit longer time can we reduce that time gap.
ReplyDeleteI like it very much, Also I really enjoyed reading the post.
ReplyDeleteFantastic post! This could aid lots of people find out about this matter.
ReplyDeleteI want to ask why i get a weird file content just like:[B@ce0197 , the jsp code is:
ReplyDeletebyte[] contentBytes = null;
try {
contentBytes = IOUtils.toByteArray(in);
out.println("contentByte :" + contentBytes);
} finally {
IOUtils.closeQuietly(in);
}
@quiming
ReplyDeleteYou code works fine. What it does is:
1. get array of bytes from input stream
2. print the array to the console
Bytearray is not a string. If you print out a string to the console, you get its content. When you print out a bytearray, you get it's "visual representation" (you can think about it as a "kind of pointer" to the variable).
If you need to load string instead of array, use IOUtils.toArray(). If you need to read bytearray and then show its string representation, convert it like I did here:
InputStream in = null;
byte[] contentBytes = null;
try {
in = new FileInputStream("test.txt");
contentBytes = IOUtils.toByteArray(in);
System.out.println(new String(contentBytes));
} catch (IOException e) {
e.printStackTrace();
} finally {
IOUtils.closeQuietly(in);
}
Hope it helps.
this is excellent. thank you very much.
ReplyDeleteArticles are created to express different body of knowledge. That is why I admire writers who are passionate of doing such incredible job. I salute you guys. By the way, I like you post for it is specifically talk about current issues and technicalities in life. I look forward for your subsequent post.I look forward for your next article.Thanks Marks Liferay Blog
ReplyDeleteYour blog helped me alot.. i was looking for this since 3 days.. and was trying different things by looking into GAE's tutorial on file upload but wasn't helpful to upload larger files( > 1mb).. Very Very thanks... good work and keep posting things like this
ReplyDeleteHello,
ReplyDeleteMay i ask you a question, in which directory of the web server do you upload your file to?
In my case, if i want to upload my file to a directory named /WebContent/File in my application server, how do i do it?
Thanks!
Everything works fine in google server emulator.
ReplyDeleteBut not at google hos -
java.lang.NullPointerException
at java.io.File.(File.java:276)
at org.apache.commons.fileupload.disk.DiskFileItem.getTempFile(DiskFileItem.java:565)
at org.apache.commons.fileupload.disk.DiskFileItem.getOutputStream(DiskFileItem.java:510)
at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:366)
at org.apache.commons.fileupload.servlet.ServletFileUpload.parseRequest(ServletFileUpload.java:126)
....
@Anonymous
ReplyDelete> In my case, if i want to upload my file to a
> directory named /WebContent/File in my
> application server, how do i do it?
Unfortunately, the answer is: you don't. GAE does not let you write to the filesystem.
@Henry
ReplyDeleteLooks like you are using DiskFileItem which tries to write to the filesystem, which GAE forbids.
Please, see doPost in my example for hint how to avoid the problem.
The curious thing is what you are saying about different behavior of dev environment and production environment.
I know there were some problems with this in the past (i.e. http://code.google.com/p/googleappengine/issues/detail?id=1655) but I thought it was fixed now.
Hi!
ReplyDeleteMay I question.
I copyed In UserDAO file have statement :
import org.fbc.shogiserver.entities.UserDetails;
import org.fbc.shogiserver.entities.UserPreferences;
can you post 2 file UserDeatail and UserPreferences ?
and
PersistenceManager pm = PMF.get().getPersistenceManager();
Can you explain for me PMF is what, and where.
Thank you very much!
@Vũ Hải Nam
ReplyDeleteThe sources are located at code.google.com. (for example, UserPreferences are here: http://code.google.com/p/shogitools/source/browse/ShogiServer/src/org/fbc/shogiserver/entities/UserPreferences.java?r=285).
PMF is a persistance manager factory. See https://developers.google.com/appengine/docs/java/datastore/jdo/overview.
The code for the class is here:
http://code.google.com/p/shogitools/source/browse/ShogiServer/src/org/fbc/shogiserver/dao/PMF.java?r=285
Best wishes,
fbc
Very helpful. Thank you very much.
ReplyDelete