Server error when passwordFormat="Hashed"

I'm trying to get hashed passwords working but setting passwordFormat="Hashed" makes the site unavailable. When I change the three web.config settings back to passwordFormat="Clear" the site works fine for both Windows and FBA user logins.
Unavailable in that the site doesn't come up at all anymore, or unavailable in that fba users can't login? If the site doesn't come up, are there any error messages (Onscreen, log file, event viewer)?
When I go to the site, the login form doesn't display. Here are the onscreen error messages I have been able to see.
Server Error in '/' Application.
The server was unable to process the request due to an internal error.  For more information about the error, either turn on IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or from the <serviceDebug> configuration behavior) on the server in order to send the exception information back to the client, or turn on tracing as per the Microsoft .NET Framework 3.0 SDK documentation and inspect the server trace logs.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.ServiceModel.FaultException: The server was unable to process the request due to an internal error.  For more information about the error, either turn on IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or from the <serviceDebug> configuration behavior) on the server in order to send the exception information back to the client, or turn on tracing as per the Microsoft .NET Framework 3.0 SDK documentation and inspect the server trace logs.

Source Error:

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Stack Trace:


[FaultException: The server was unable to process the request due to an internal error.  For more information about the error, either turn on IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or from the <serviceDebug> configuration behavior) on the server in order to send the exception information back to the client, or turn on tracing as per the Microsoft .NET Framework 3.0 SDK documentation and inspect the server trace logs.]
   Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.ReadResponse(Message response) +1164261
   Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecurityToken rst, RequestSecurityTokenResponse& rstr) +73
   Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecurityToken rst) +36
   Microsoft.SharePoint.SPSecurityContext.SecurityTokenForContext(Uri context, Boolean bearerToken, SecurityToken onBehalfOf, SecurityToken actAs, SecurityToken delegateTo) +26758801
   Microsoft.SharePoint.SPSecurityContext.SecurityTokenForFormsAuthentication(Uri context, String membershipProviderName, String roleProviderName, String username, String password, Boolean isPersistent) +26754684
   Microsoft.SharePoint.IdentityModel.Pages.FormsSignInPage.GetSecurityToken(Login formsSignInControl) +210
   Microsoft.SharePoint.IdentityModel.Pages.FormsSignInPage.AuthenticateEventHandler(Object sender, AuthenticateEventArgs formAuthenticateEvent) +123
   System.Web.UI.WebControls.Login.AttemptLogin() +152
   System.Web.UI.WebControls.Login.OnBubbleEvent(Object source, EventArgs e) +124
   System.Web.UI.Control.RaiseBubbleEvent(Object source, EventArgs args) +70
   System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument) +29
   System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +2981


Version Information: Microsoft .NET Framework Version:2.0.50727.5485; ASP.NET Version:2.0.50727.5483  
And then with <serviceDebug> added to the Security Token web.config:
<head> 
<title>IIS 7.5 Detailed Error - 500.19 - Internal Server Error</title> 
<style type="text/css"> 
<!-- 
body{margin:0;font-size:.7em;font-family:Verdana,Arial,Helvetica,sans-serif;background:#CBE1EF;} 
code{margin:0;color:#006600;font-size:1.1em;font-weight:bold;} 
.config_source code{font-size:.8em;color:#000000;} 
pre{margin:0;font-size:1.4em;word-wrap:break-word;} 
ul,ol{margin:10px 0 10px 40px;} 
ul.first,ol.first{margin-top:5px;} 
fieldset{padding:0 15px 10px 15px;} 
.summary-container fieldset{padding-bottom:5px;margin-top:4px;} 
legend.no-expand-all{padding:2px 15px 4px 10px;margin:0 0 0 -12px;} 
legend{color:#333333;padding:4px 15px 4px 10px;margin:4px 0 8px -12px;_margin-top:0px; 
 border-top:1px solid #EDEDED;border-left:1px solid #EDEDED;border-right:1px solid #969696; 
 border-bottom:1px solid #969696;background:#E7ECF0;font-weight:bold;'.]
   System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) +10266458
   System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) +539
   Microsoft.IdentityModel.Protocols.WSTrust.IWSTrustContract.Issue(Message message) +0
   Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecurityToken rst, RequestSecurityTokenResponse& rstr) +61
   Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecurityToken rst) +36
   Microsoft.SharePoint.SPSecurityContext.SecurityTokenForContext(Uri context, Boolean bearerToken, SecurityToken onBehalfOf, SecurityToken actAs, SecurityToken delegateTo) +26758801
   Microsoft.SharePoint.SPSecurityContext.SecurityTokenForFormsAuthentication(Uri context, String membershipProviderName, String roleProviderName, String username, String password, Boolean isPersistent) +183
   Microsoft.SharePoint.IdentityModel.Pages.FormsSignInPage.GetSecurityToken(Login formsSignInControl) +210
   Microsoft.SharePoint.IdentityModel.Pages.FormsSignInPage.AuthenticateEventHandler(Object sender, AuthenticateEventArgs formAuthenticateEvent) +123
   System.Web.UI.WebControls.Login.AttemptLogin() +152
   System.Web.UI.WebControls.Login.OnBubbleEvent(Object source, EventArgs e) +124
   System.Web.UI.Control.RaiseBubbleEvent(Object source, EventArgs args) +70
   System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument) +29
   System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +2981


Version Information: Microsoft .NET Framework Version:2.0.50727.5485; ASP.NET Version:2.0.50727.5483   
I'll post the section I located in log for the web application in a separate reply.
On my test site I have the automatic sign-in with mixed authentication package deployed spautomaticsignin on CodePlex When I have it set to prompt me for an FBA user login, I do get to the Sign-In form, but after I enter the credentials I get the error message mentioned in the first part of my previous reply. Here is the section from the logs for my FBA webapplication in that scenario.
10/11/2014 06:37:43.32  w3wp.exe (0x1E44)                           0x1A24  SharePoint Foundation           General                         8e2s    Medium      Unknown SPRequest error occurred. More information: 0x80070005  61948afb-c744-4b2a-9b29-8bd2cb44a1ee
10/11/2014 06:37:43.33  w3wp.exe (0x1E44)                           0x1A24  SharePoint Foundation           Claims Authentication           fsq7    High        Request for security token failed with exception: System.ServiceModel.ProtocolException: The content type text/html; charset=utf-8 of the response message does not match the content type of the binding (application/soap+msbin1). If using a custom encoder, be sure that the IsContentTypeSupported method is implemented properly. The first 1024 bytes of the response were: '<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">  <html xmlns="http://www.w3.org/1999/xhtml">  <head>  <title>IIS 7.5 Detailed Error - 500.19 - Internal Server Error</title>  <style type="text/css">  <!--  body{margin:0;font-size:.7em;font-family:Verdana,Arial,Helvetica,sans-serif;background:#CBE1EF;}  code{margin:0;color:#006600;font-size:1.1em;font-weight:bold;} ... 61948afb-c744-4b2a-9b29-8bd2cb44a1ee
10/11/2014 06:37:43.33* w3wp.exe (0x1E44)                           0x1A24  SharePoint Foundation           Claims Authentication           fsq7    High        ... .config_source code{font-size:.8em;color:#000000;}  pre{margin:0;font-size:1.4em;word-wrap:break-word;}  ul,ol{margin:10px 0 10px 40px;}  ul.first,ol.first{margin-top:5px;}  fieldset{padding:0 15px 10px 15px;}  .summary-container fieldset{padding-bottom:5px;margin-top:4px;}  legend.no-expand-all{padding:2px 15px 4px 10px;margin:0 0 0 -12px;}  legend{color:#333333;padding:4px 15px 4px 10px;margin:4px 0 8px -12px;_margin-top:0px;   border-top:1px solid #EDEDED;border-left:1px solid #EDEDED;border-right:1px solid #969696;   border-bottom:1px solid #969696;background:#E7ECF0;font-weight:bold;'. ---> System.Net.WebException: The remote server returned an error: (500) Internal Server Error.     at System.Net.HttpWebRequest.GetResponse()     at System.ServiceModel.Channels.HttpChannelFactory.Htt...  61948afb-c744-4b2a-9b29-8bd2cb44a1ee
10/11/2014 06:37:43.33* w3wp.exe (0x1E44)                           0x1A24  SharePoint Foundation           Claims Authentication           fsq7    High        ...pRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)     --- End of inner exception stack trace ---    Server stack trace:      at System.ServiceModel.Channels.HttpChannelUtilities.ValidateRequestReplyResponse(HttpWebRequest request, HttpWebResponse response, HttpChannelFactory factory, WebException responseException, ChannelBinding channelBinding)     at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)     at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)     at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)     at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operati...  61948afb-c744-4b2a-9b29-8bd2cb44a1ee
10/11/2014 06:37:43.33* w3wp.exe (0x1E44)                           0x1A24  SharePoint Foundation           Claims Authentication           fsq7    High        ...on, Object[] ins, Object[] outs, TimeSpan timeout)     at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)     at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)    Exception rethrown at [0]:      at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)     at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)     at Microsoft.IdentityModel.Protocols.WSTrust.IWSTrustContract.Issue(Message message)     at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecurityToken rst, RequestSecurityTokenResponse& rstr)     at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecur...  61948afb-c744-4b2a-9b29-8bd2cb44a1ee
10/11/2014 06:37:43.33* w3wp.exe (0x1E44)                           0x1A24  SharePoint Foundation           Claims Authentication           fsq7    High        ...ityToken rst)     at Microsoft.SharePoint.SPSecurityContext.SecurityTokenForContext(Uri context, Boolean bearerToken, SecurityToken onBehalfOf, SecurityToken actAs, SecurityToken delegateTo)   61948afb-c744-4b2a-9b29-8bd2cb44a1ee
10/11/2014 06:37:43.33  w3wp.exe (0x1E44)                           0x1A24  SharePoint Foundation           Claims Authentication           8306    Critical    An exception occurred when trying to issue security token: The content type text/html; charset=utf-8 of the response message does not match the content type of the binding (application/soap+msbin1). If using a custom encoder, be sure that the IsContentTypeSupported method is implemented properly. The first 1024 bytes of the response were: '<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">  <html xmlns="http://www.w3.org/1999/xhtml">  <head>  <title>IIS 7.5 Detailed Error - 500.19 - Internal Server Error</title>  <style type="text/css">  <!--  body{margin:0;font-size:.7em;font-family:Verdana,Arial,Helvetica,sans-serif;background:#CBE1EF;}  code{margin:0;color:#006600;font-size:1.1em;font-weight:bold;}  .config_source code{font-size... 61948afb-c744-4b2a-9b29-8bd2cb44a1ee
10/11/2014 06:37:43.33* w3wp.exe (0x1E44)                           0x1A24  SharePoint Foundation           Claims Authentication           8306    Critical    ...:.8em;color:#000000;}  pre{margin:0;font-size:1.4em;word-wrap:break-word;}  ul,ol{margin:10px 0 10px 40px;}  ul.first,ol.first{margin-top:5px;}  fieldset{padding:0 15px 10px 15px;}  .summary-container fieldset{padding-bottom:5px;margin-top:4px;}  legend.no-expand-all{padding:2px 15px 4px 10px;margin:0 0 0 -12px;}  legend{color:#333333;padding:4px 15px 4px 10px;margin:4px 0 8px -12px;_margin-top:0px;   border-top:1px solid #EDEDED;border-left:1px solid #EDEDED;border-right:1px solid #969696;   border-bottom:1px solid #969696;background:#E7ECF0;font-weight:bold;'..    61948afb-c744-4b2a-9b29-8bd2cb44a1ee
10/11/2014 06:37:43.33  w3wp.exe (0x1E44)                           0x1A24  SharePoint Foundation           Monitoring                      b4ly    Medium      Leaving Monitored Scope (Request (POST:http://ca7078:41931/_forms/default.aspx?ReturnUrl=%2fsw%2f_layouts%2fAuthenticate.aspx%3fSource%3d%252Fsw%252FSitePages%252FJudicialReview%252Easpx&Source=%2Fsw%2FSitePages%2FJudicialReview.aspx)). Execution Time=22.1438463865065    61948afb-c744-4b2a-9b29-8bd2cb44a1ee
Hopefully something there sheds some light on the issue. I'm not sure about that <serviceDebug> message in my previous reply. I think I caused that by trying implement that debugging capability incorrectly.

I set passwordFormat back to Clear, logged in as a windows user, changed passwordFormat back to Hashed, then tried to navigate to a different page on the site. I encountered the same unknown error message as before, but I noticed this in the logs:
SPSecurityTokenService.Issue() failed: System.Configuration.ConfigurationErrorsException: Configured settings are invalid: Hashed passwords cannot be retrieved. Either set the password format to different type, or set supportsPasswordRetrieval to false.

I changed the password retrieval to false. But then I get an error again and the site url goes to http://ca7078:41931/_layouts/error.aspx. I get this error whether passwordFormat is Hashed or Clear.

Thanks for the prompt response. I wasn't expecting that on a Friday night!
I may have figured this out, though I'm not exactly sure. On my test site I had the FBAMembershipProvider listed in the providers for all three web.config files and also in the machine.config file. But I had added it the machine.config yesterday when I first started trying to get hashed passwords to work. As a result I was getting an error about the FBAMembershipProvider already being added:

High SOAP exception: System.Configuration.ConfigurationErrorsException: The entry 'FBAMembershipProvider' has already been added. (C:\inetpub\wwwroot\wss\VirtualDirectories\41931\web.config line 215) at System.Web.Security.Membership.Initialize() at System.Web.Security.Membership.get_Provider() at Microsoft.SharePoint.Utilities.SPUtility.FormatAccountName(String user) at Microsoft.SharePoint.SPGlobal.CreateSPRequestAndSetIdentity(SPSite site, String name, Boolean bNotGlobalAdminCode, String strUrl, Boolean bNotAddToContext, Byte[] UserToken, String userName, Boolean bIgnoreTokenTimeout, Boolean bAsAnonymous) at Microsoft.SharePoint.SPRequestManager.GetContextRequest(SPRequestAuthenticationMode authenticationMode) at Microsoft.SharePoint.Administration.SPFarm.get_RequestA... 9fb066ba-e5ed-4e73-950d-a78d7be6a275

I removed the FBAMembershipProvider line from the machine.config file. Set enablePasswordRetrieval="false" and passwordFormat="Hashed" in all three web.config files. Now I can create a new user, the new user's password is hashed in the aspnetdb and the user can sign into the site. For an existing user it appears I have to go into the aspnetdb and change the password format field from "0" to "1".

Then I added the recover password webpart back to the site and it works fine as well. I failed to restart iis during some of my previous config changes, which probably contributed to the errors I was encountering. My current MembershipProvider settings in all three web.config files is:
   <membership defaultProvider="i">
      <providers>
        <add name="i" type="Microsoft.SharePoint.Administration.Claims.SPClaimsAuthMembershipProvider, Microsoft.SharePoint, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" />
        <add name="FBAMembershipProvider" type="System.Web.Security.SqlMembershipProvider, System.Web, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" applicationName="/" connectionStringName="FBADB" enablePasswordReset="true" enablePasswordRetrieval="false" passwordFormat="Hashed" requiresQuestionAndAnswer="false" requiresUniqueEmail="true" /> 
      </providers>
    </membership>
All the FBA features seem to be working. Hopefully I can get the same outcome on the live site.
Great to hear you got it working! Yes, I think these changes should also fix the live site. The one thing i'll mention is that the "i" membership provider only belongs in the web application web.config's. It should not be in the Security Token web.config.